The design of fluxgate magnetometers is typically a nonlinear multi-objective optimization problem. Different objectives often conflict with each other, and sometimes an optimal Fluxgate Magnetometer Sensor (FMS) performance is difficult to achieve. The sensitivity of the sensor decreases with an increase of noise level while trying to reduce the sensor dimension. Hence, there is need for a systematic optimization approach for FMS design to find its optimum performance. The combined modified multi-objective Firefly Optimization Algorithm (FOA) and systematic optimization approach is suggested to improve FMS’s design in this research by simultaneously optimizing the sensitivity and noise of a FMS while the sensor core, pick-up coil, and detection circuit are minimized. The developed model allowed improved sensitivity of 86.65%, reduction of noise level by 59.97% while still keeping the sensor size small by 14.29%. Keywords: Fluxgate magnetometer sensor, noise, sensitivity, firefly optimization algorithm. DOI : 10.7176/ISDE/10-4-03 Publication date :May 31 st 2019