A spectral integral method is presented for electromagnetic scattering from dielectric and perfectly electric conducting (PEC) objects with a closed boundary embedded in a layered medium. Two-dimensional layered medium Green's functions are computed adaptively by using Gaussian quadratures. The singular terms in the Green's functions and the non-smooth terms in their derivatives are handled appropriately to achieve exponential convergence. Numerical results, compared with the ones obtained by using other methods, demonstrate the spectral accuracy and high efficiency of the proposed method. They also confirm that the spectral integral method (SIM) is applicable to concave objects.