Object detectors usually required manual design of sliding windows, but they usually faced the shortage of feature generalization capabilities. Even the development of modern mainstream anchor base detectors also required artificial design scale and ratio of the anchor. This is a complicated and tedious process, and the quality of final effect needs to be determined through multiple experiments. In this paper, we present a new structure of object detection that choosing anchor free detection method, treat the object as entirety with advanced semantic features, the convolution predictor scans all regions and activated by these advanced features. Compared to anchor base method, the detector's activated object is the high-level feature of the target, and therefore it is not vulnerable to complex and changeable background environments. In addition, we set the center of gravity of instance as the key point, which effectively avoids the shift of subject area of interest caused by the special shape pull; using the center of gravity combined with the height and width information provided by the anchor box, resulting asymmetric Gaussian mask can play better effect in crowded scenarios; add FPN structure to the detection network aim to enhanced the generalization ability of the detector for objects of different scales. By designing a simple structured object detection network MFGP, we achieved considerable performance comparable to complex structured object detector, which has a good performance on the object detection dataset COCO and the pedestrian detection dataset CityPersons. The best neutralization of accuracy and speed was achieved on the COCO dataset, reaching 41.2% AP when 18 FPS; on the reasonable subset of the CityPersons dataset, it reached 10.9% MR−2 without significantly increasing the test time.