With the rapid introduction of autonomous agents into everyday tasks, concerns about agent alignment to human moral norms are becoming increasingly prominent. The widespread adoption of reinforcement learning (RL) in autonomous decision-making has intensified the challenge of ensuring that these algorithms align agents' behaviour with moral and ethical values. While most common approaches to value alignment focus on the learning algorithms agents use, a recently introduced algorithm called ethical embedding shifts the focus to designing ethical environments rather than modifying agents' learning algorithms. This transition from an agent-centred view to an environment-centred perspective opens new opportunities for developing safe and trustworthy AI agents. This project aims to advance this line of research by exploring environment design as a means to create non-manipulable learning environments, where the environment itself guides agents toward ethical behaviour, regardless of the learning algorithm employed. Furthermore, as a novel contribution to previous work, we integrate this approach with state-of-the-art deep reinforcement learning algorithms, enabling the application of these techniques in realistic environments suitable for training agents that operate in the real world.