In underwater acoustic sensor networks (UASNs), the unpredictable nature of the underwater acoustic channel presents significant challenges for reliable communication. Traditional medium access control (MAC) protocols, designed for more stable terrestrial environments, struggle to perform effectively in these circumstances. This paper evaluates the performance of UW-ALOHA-Q, a reinforcement learning (RL)-based MAC protocol designed for UASNs, focusing on its adaptability and performance in the face of the underwater channel's inherent unreliability—an aspect not thoroughly examined in prior evaluations. Utilizing the DESERT Underwater simulator, we investigate the impact of channel conditions on the effectiveness of UW-ALOHA-Q’s learning mechanism. Our results show that UW-ALOHA-Q outperforms conventional protocols such as ALOHA-CS and TDMA in terms of channel utilization, but faces challenges in achieving convergence in highly unreliable channel conditions. Our study underscores the potential of RL-based MAC protocols in enhancing the robustness and efficiency of UASNs, while also identifying critical areas for further research in RL methodology to address the unique challenges of underwater environments.