This paper studies the impact of mobility management and seamless handover mechanisms on the perceived quality of video streaming applications. A seamless handoff scheme is proposed that incorporates IEEE 802.21 Media Independent Handover framework and a QoE-driven rate adaptation scheme, for both scalable and single layer coding. Quality of Experience (QoE) has been considered as an important parameter for network selection. The proposed scheme has been implemented and evaluated in a real test-bed network. Objective and subjective quality evaluation measurements for both H.264/AVC and H.264/SVC have been carried out. Through experimentation, QoE driven mobility of video streaming can be better maintained with H.264/SVC under the combined seamless handoff and rate adaptation functionality.