SSI and site amplification effects are investigated as influences on the seismic fragility of existing reinforced concrete moment-resisting frame and dual frame-wall system buildings supported on shallow foundations without interconnecting beams. We build upon a holistic methodology that accounts for site amplification and soil-foundation-structure interaction effects using a modular approach. We calculate fragility curves based on nonlinear dynamic analyses for various building structural typologies and geometries, infill conditions, code provisions, and soil profile materials and dynamic characteristics. We demonstrate that site amplification during earthquakes may significantly increase the fragility of the soil-foundation-structure system, which is reflected in its vulnerability. Moreover, SSI is especially prevalent for buildings on soft soil profiles and might modify their fragility. We propose a modular method to include site amplification and/or soil-structure interaction effects in a large-scale earthquake vulnerability assessment using fragility modifiers, which we express using an easy-to-code equation form.