We propose a novel hybrid mode interferometer (HMI) leveraging the interference of hybridized TE–TM modes in a silicon-on-insulator (SOI) waveguide integrated with a GeSe phase change material (PCM) layer. The SOI waveguide’s dimensions are optimized to support the hybridization of the fundamental transverse magnetic (TM0) and the first higher transverse electric (TE1) mode. This design allows for efficient and nearly equal power coupling between these two modes, resulting in high-contrast interference when starting from the amorphous PCM state. The PCM’s phase transition induces a differential change in the modal effective index, enabling high-contrast transmittance modulation. Our numerical simulations demonstrate a multilevel transmission with a high contrast of nearly 14 dB when the amorphous region’s length is varied incrementally, enabling multi-bit storage. The transmittance is maximized in the fully crystalline state with an insertion loss below 0.1 dB. The HMI can also operate as a quasi-pure phase shifter when partially amorphized, making it suitable for Mach–Zehnder interferometers. These characteristics make the proposed device a promising candidate for applications in photonic memories and neuromorphic computing.