In this paper, we present a new end-to-end protocol, namely Scalable Streaming Video Protocol (SSVP), which operates on top of UDP and is optimized for unicast video streaming applications. SSVP employs Additive Increase Multiplicative Decrease (AIMD)-based congestion control and adapts the sending rate by properly adjusting the inter-packet-gap (IPG). The smoothness-oriented modulation of AIMD parameters and IPG adjustments reduce the magnitude of AIMD oscillation and allow for smooth transmission patterns, while TCP-friendliness is maintained. Our experimental results demonstrate that SSVP eventually adapts to the vagaries of the network and achieves remarkable performance on real-time video delivery. In the event where awkward network conditions impair the perceptual video quality, we investigate the potential improvement via a layered adaptation mechanism that utilizes receiver buffering and adapts video quality along with long-term variations in the available bandwidth. The adaptation mechanism sends a new layer based on explicit criteria that consider both the available bandwidth and the amount of buffering at the receiver, preventing wasteful layer changes that have an adverse effect on user-perceived quality. Quantifying the interactions of SSVP with the specific adaptation scheme, we identify notable gains in terms of video delivery, especially in the presence of limited bandwidth.