Flexible Network Service composition is a fundamental enabler for the design of 6G networks, where network services are modeled as ordered Service Function Chains (SFCs) with heterogeneous Virtual Network Functions (VNFs). However, dynamic traffic generated by end users and dynamic network resource infrastructure utilization make online context-aware and resource-efficient SFC composition challenging. While Deep Reinforcement Learning (DRL) has been explored for this task, the multimodal nature of traffic and the variable-length inputs limit achievable performance. To address these challenges, we propose an attention-based actor–critic framework that integrates Transformer self-attention and encoding to capture variable-length SFC states and inter-VNF dependencies. The learned representations are then used by an actor–critic policy to sequentially select resource-aware composition actions, enabling adaptive and efficient service chain construction under dynamic network conditions. Extensive simulations show that our proposed transformer-augmented actor-critic DRL achieves faster policy convergence, lower bandwidth and computational resource consumption, and higher deadline satisfaction rates compared to state-of-the-art baselines.