In this paper, we propose a reinforcement learning-based adaptive modulation and coding scheme for underwater communications; more specifically, based on the network states such as the quality of service requirement of the sensing message, the previous transmission quality, and the energy consumption. This scheme applies reinforcement learning to choose the modulation and coding policy in a dynamic underwater communication system. We provide the performance bound of this scheme and perform experiments in both pool and sea environments. The experimental data were collected and post-processed. Compared with the benchmark schemes, this scheme can improve the throughputs and reduce the BER with less energy consumption.