We study a simple model of algorithmic collusion in which Q-learning algorithms are designed in a strategic fashion. We let players (\textit{designers}) choose their exploration policy simultaneously prior to letting their algorithms repeatedly play a prisoner's dilemma. We prove that, in equilibrium, collusive behavior is reached with positive probability. Our numerical simulations indicate symmetry of the equilibria and give insight for how they are affected by a parameter of interest. We also investigate general profiles of exploration policies. We characterize the behavior of the system for extreme profiles (fully greedy and fully explorative) and use numerical simulations and clustering methods to measure the likelihood of collusive behavior in general cases.