Flame retarded and smoke suppressed vinyl ester resin (VER) were prepared through ammonium polyphosphate (APP) coupled with different smoke suppressants. The flame retardancy of these composites was tested by the limiting oxygen index and UL‐94 tests. The typical combustion parameters including heat release rate (HRR), peak of HRR (p‐HRR), total heat release, smoke production rate, and total smoke production were detected using a cone calorimeter. The smoke suppressants CaCO3, ZnMoO4, Cu2O, and Fe2O3 show different effect on flame retardancy and smoke suppression of VER composites when they are coupled with APP, the synergistic action of APP and CaCO3 is more effective on decreasing the HRR and smoke release rate than several other smoke suppressants. This is attributed to the fact that CaCO3 could promote the formation of a dense carbon layer with high thermal stability and anti‐oxidation property, which could act as an effective physical barrier. The flame retardant performances and mechanisms of APP and CaCO3 were evaluated and analyzed at length by thermogravimetric coupled with a Fourier transform infrared spectrometer (FTIR), scanning electron microscopy, FTIR, and X‐ray diffraction. POLYM. ENG. SCI., 60: 314–322, 2019. © 2019 Society of Plastics Engineers