Dialogue has long been used in human society to explain seemingly opaque concepts. In this paper we focus on how to better explain training models for neural networks, to entertain as well as inform. We present a multi-agent argumentation-based dialogue system to generate human understandable dialogue to explain backpropagation. The system incorporates a model of agent personality and introduces social elements between agents to produce characterful discussion. Natural language templates are used to render utterances in English.