Abstract The rim seal is used to prevent mainstream ingestion to the gap between the vane of a turbine and its blade. In this article, the dolphin lip with a hook configuration and a large seal cavity with hook structures are designed based on the high-pressure turbine datum single shark lip rim seal configuration. The sealing effect and parameters of the flow field are measured by an experiment method and a numerical simulation is used to explain the mechanism. For three configurations, the effect of the leakage slot vortex on the efficiency of the seal and the influence of leakage vortex, generated by the interaction between purge flow and mainstream flow, are discussed in depth. The result shows that the reverse vortex formed by the dolphin lip rim seal with hook structure will increase the sealing efficiency. The seal configuration with a large cavity improves sealing efficiency to a greater extent than the datum structure. At different purge flow rates and with unequal seal structures, the purge flow produces three types of leakage vortices in the passage. Besides, the seal configuration with dolphin lip produces a Kelvin-Helmholtz instability at the interface of the purge and the mainstream flows at a low purge flow rate to induce new leakage vortex branches in the passage of the blade.