General Relativity predicts only two tensor polarization modes for gravitational waves while at most six possible polarization modes are allowed in general metric theory of gravity. The number of polarization modes is determined by the specific modified theory of gravity. Therefore, the determination of polarization modes can be used to test gravitational theory. We introduce a concrete data analysis pipeline for a space-based detector such as LISA to detect the polarization modes of gravitational waves. This method can be used for monochromatic gravitational waves emitted from any compact binary system with known sky position and frequency to detect mixtures of tensor and extra polarization modes. We use the source J0806.3+1527 with one-year simulation data as an example to show that this approach is capable of probing pure and mixed polarizations without knowing the exact polarization modes. We also find that the ability of detection of extra polarization depends on the gravitational wave source location and the amplitude of non-tensorial components.

29 pages, 12 figures, 1 table. (PRD received)