Viewport-Dependent Streaming (VDS) is a preferred way in practice to deliver the omnidirectional videos, of which a High-Quality (HQ) scale is applied for the content in current viewport but a Low-Quality (LQ) scale elsewhere. Quality adaptation or refinement happens after users stabilize their fixations to a new viewport. In this article, we formulate this as a perceptually optimized quality adaptation problem to maximize the Quality of Experience (QoE) for the refinement from a LQ scale to another HQ level within a specific duration under the given network constraint. With our developed perceptual model considering the adaptation quality for VDS of omnidirectional videos, we first provide baseline solutions numerically, demonstrating the noticeable subjective improvements of model-driven solution against the heuristic selection based approach. We also propose a novel viewport prediction algorithm based on the Hidden Markov Model (HMM), and experimental results show that it significantly outperforms the relevant methods with better prediction accuracy. We then improve the adaptation strategy with proposed viewport prediction-based data prefetching, leading to better visual perception than the baseline system at the same bandwidth constraint. Generally, prefetching the content of predicted next viewport one second ahead of its playback time, would lead to more than 8% Bjontegaard Delta Rate (BD-Rate) gain.