Optimal Transmission Switching (OTS) reduces generation cost by strategically opening transmission lines, but its mixed-integer linear program (MILP) formulation scales poorly for large-scale transmission networks. Reinforcement learning (RL) offers a computationally efficient alternative, but existing RL-based OTS approaches rely on soft penalties that permit physical constraint violations. This paper presents a comparison between an RL framework and an MILP-based optimization method for OTS. Case studies were carried out on the IEEE RTS-96 24-bus system; results show that the agent was able to produce near-optimal solutions at low switching budgets and tended to yield suboptimal solutions at high switching budgets. However, the RL agent was able to generate feasible solutions two-to-three orders of magnitude faster than the optimization solver.