In this paper an effective hybrid FAT-SGO approach is proposed for islanding detection of distributed generation (DG) system. The proposed approach is the joint implementation of Feedback Artificial Tree (FAT) and Shell Game Optimization (SGO) named as FAT-SGO technique. Reducing the non-detection zone (NDZ) as near as possible and keep the output power quality unmovable is main contribution of this paper. Furthermore, this method solves the issue of establishing detection thresholds inherent in existing methods. The proposed strategy uses the rate of change of frequency (ROCOF) in DG destination location is utilized as input sets of FAT system for intelligent islanding detection. Here, FAT is trained by SGO, which extracts the different intrinsic characteristics among islanding and grid disturbance. With the extracted characteristics, the FAT method is used for classifying the disturbances in islanding and grid. For authenticating the feasibility of this strategy is authorized through various conditions and different conditions of load, switching operation, and network. The simulation of the proposal is done in MATLAB SIMULINK and the performance in islanding and non-islanding events was studied. Statistic analysis of proposed and existing methods of mean, median and standard deviation is analyzed. DG performance is assessed by comparative analysis with current techniques.