We have investigated the effect of an ultra-thin Ta insertion in the CoFeB (CoFeB/Ta/CoFeB) free layer (FL) on magnetic and tunneling magnetoresistance (TMR) properties of a CoFeB-MgO system with perpendicular magnetic anisotropy (PMA). It is found that the critical thickness (tc) to sustain PMA is doubled (tc = 2.6 nm) in Ta-inserted CoFeB FL as compared to single CoFeB layer (tc = 1.3 nm). While the effective magnetic anisotropy is found to increase with Ta insertion, the saturation magnetization showed a slight reduction. As the CoFeB thickness increasing, the thermal stability of Ta inserted structure is significantly increased by a factor of 2.5 for total CoFeB thickness less than 2 nm. We have observed a reasonable value of TMR for a much thicker CoFeB FL (thickness = 2-2.6 nm) with Ta insertion, and without significant increment in resistance-area product. Our results reveal that an ultra-thin Ta insertion in CoFeB might pay the way towards developing the high-density memory devices with enhanced thermal stability.