Edge computing has promoted various applications driven by artificial intelligence (AI). However, upgrading AI models during system operation may change resource and performance features. Then, the service management controller (SMC) faces an unprecedented environmental condition and has limited prior knowledge, resulting in high probabilities of policy mismatches. With the proliferation of AI applications, it is an urgent necessity that SMCs can adapt to different conditions to ensure quality of service (QoS) and resource efficiency. Therefore, this paper studies the problem of dynamic edge AI service adaptation and formulates it as a multi-task scenario adaptation problem. After that, we proposed an approach based on off-policy meta-reinforcement learning and digital twin (DT) technology. The DT system emulates a set of encountered conditions, and a meta-policy is obtained by interacting with these DTs. The executed policy is initialized as the meta-policy once AI models are upgraded. Then, it adapts to new service conditions by drawing salient information from limited transition contexts collected from a newly encountered environmental condition. Simulation results reveal that our approach can optimize QoS and adapt to different service situations.