The problems associated with open-ended group discussion are well-documented in sociology research. We seek to alleviate these issues using technology that autonomously serves as a discussion moderator. Building on top of an extensible framework called Diplomat, we develop a "conversational agent", ArbiterBot to promote efficiency, fairness, and professionalism in otherwise unstructured discussions. To evaluate the effectiveness of this agent, we recruited university students to participate in a study involving a series of prompted discussions over the Slack messenger app. The results of this study suggest that the conversational agent is effective at balancing contributions across participants, encouraging a timely consensus and promoting a higher coverage of topics. We believe that the results motivate further investigation into how conversational agents can be used to improve group discussion and cooperation.