Effects of electron correlations on a two dimensional quantum spin Hall (QSH) system are studied. We examine possible phases of a generalized Hubbard model on a bilayer honeycomb lattice with a spin-orbit coupling and short range electron-electron repulsions at half filling, based on the slave rotor mean-field theory. Besides the conventional QSH phase and a broken-symmetry insulating phase, we find a new phase, a fractionalized quantum spin Hall phase, where the QSH effect arises for fractionalized spinons which carry only spin but not charge. Experimental manifestations of the exotic phase and effects of fluctuations beyond the saddle point approximation are also discussed.

6 pages, 2 figures; v3) discussions on the stability of the edge modes added