Enactive and ecological approaches to cognitive science both claim a "mutuality" between agents and their environments - that they have a complementary nature and should be addressed as a single whole system. Despite this apparent agreement, each offers criticisms of the other on precisely this point - enactivists claiming that ecological psychologists over-emphasize the environment, while the complementary criticism, of agent-centered constructivism, is leveled by ecological psychologists at enactivists. In this paper I suggest that underlying the confusion between the two approaches is the complexity of agency, which comes in different forms, at different scales or levels of analysis. Cognitive science has not theorized the relationship between these different forms in a sufficiently disciplined manner, and a task therefore remains of finding a way to map the complex territory of agency.