Currently, the discussions and investigations for the vacuum energy are drawing great both theoretical and experimental attention. The vacuum states of variety of fields, subject to special boundary conditions, may contribute to non-trivial macroscopic vacuum energy, i.e., the Casimir effect, which become an interdisciplinary subject and plays an important role in a variety of fields of physics. We adopt Schwingers source theory and study the quantization of gravitation contributions to the Casimir effect, i.e., the gravitational Casimir effect, in inspiraling neutron star binaries with wide separation of 109 m. By considering gravitoelectromagnetism (GEM) arising from the spiral-in orbital motion and evaluating the gravitoelectromagnetic contributions to the vacuum energy of gravitons radiated during the orbital decay, we demonstrate that, when the radial separation of the system decay a distance L, the GEM results in a small Casimir correction to the gravitational vacuum energy, which contributes to an attractive gravitational Casimir force to the binary, in addition to the gravitational force. We also discuss the possible detections for such gravitational Casimir effect from both the future space-based gravitational wave observatories and cosmological observations.