Simulation works are reported for the optimization of alignment mark geometries to enhance the magnitude of alignment signal and to improve precision of alignment and overlay process. Modeling of diffraction of electromagnetic wave on alignment mark is based on rigorous coupled-wave analysis (RCWA). This simulation allows calculating of overlay signal and optimizing of alignment mark. Using of RCWA approach significantly decreases computational time and required memory size comparing with FDTD. Overlay signal is usually measured using one of the diffraction orders. Thus the great advantage of using RCWA instead of FDTD is possibility to find amplitude of this diffracted order directly.