Controlling agent behavior is a fundamental challenge across diverse domains within artificial intelligence and robotics. The central idea of this dissertation is that shared knowledge can be used as a powerful tool to control AI agents’ behavior. This dissertation explores the utilization of shared knowledge in constructing coherent narratives and enhancing the expression of shared knowledge in Reinforcement Learning agents. In this dissertation, I first investigate the utilization of shared knowledge for constructing narratives by developing a story-generation agent that emulates the cognitive processes of how human readers create detailed mental models, referred to as the “reader model”, which they use to understand and interpret stories with shared knowledge. Employing the reader model has resulted in the generation of significantly more coherent and goal-directed stories. I also explore how to input unique constraints into the story generator allowing for the modification of the shared knowledge. Subsequently, I delve into the application of shared knowledge in controlling reinforcement learning agents through the introduction of a technique called “Story Shaping.” This technique involves the agent inferring tacit knowledge from an exemplar story and rewarding itself for actions that align with the inferred reader model. Following proposing this agent, I propose the Thespian agent to leverage the knowledge learned in this technique to adapt to the new environment under a few-shot setting. Additionally, I investigate the potential of using shared knowledge to explain behavior by examining the impact of symbolic knowledge graph-based state representation and Hierarchical Graph Attention mechanism on the decision-making process of a reinforcement learning agent. The goal of this dissertation is to create AI-driven systems that are more coherent, controllable, and aligned with human expectations and preferences, thereby fostering trust and safety in human-AI interactions. ; Ph.D.