The immersed boundary method is one of the most useful computational methods in studying fluid structure interaction. On the other hand, the Immersed Boundary method is also known to require small time steps to maintain stability when solved with an explicit method. Many implicit or approximately implicit methods have been proposed in the literature to remove this severe time step stability constraint, but none of them give satisfactory performance. In this paper, we propose an efficient semi-implicit scheme to remove this stiffness from the immersed boundary method for the Navier-Stokes equations. The construction of our semi-implicit scheme consists of two steps. First, we obtain a semi-implicit discretization which is proved to be unconditionally stable. This unconditionally stable semi-implicit scheme is still quite expensive to implement in practice. Next, we apply the small scale decomposition to the unconditionally stable semi-implicit scheme to construct our efficient semi-implicit scheme. Unlike other implicit or semi-implicit schemes proposed in the literature, our semi-implicit scheme can be solved explicitly in the spectral space. Thus the computational cost of our semi-implicit schemes is comparable to that of an explicit scheme. Our extensive numerical experiments show that our semi-implicit scheme has much better stability property than an explicit scheme. This offers a substantial computational saving in using the immersed boundary method.