The traditional method to solve nondeterministic-polynomial-time (NP)-hard optimization problems is to apply meta-heuristic algorithms. In contrast, Deep Q Learning (DQL) uses memory of experience and deep neural network (DNN) to choose steps and progress towards solving the problem. The dynamic time-division multiple access (DTDMA) scheme is a viable transmission method in visible light communication (VLC) systems. In DTDMA systems, the time-slots of the users are adjusted to maximize the spectral efficiency (SE) of the system. The users in a VLC network have different channel gains because of their physical locations, and the use of variable time-slots can improve the system performance. In this work, we propose a Markov decision process (MDP) model of the DTDMA-based VLC system. The MDP model integrates into deep Q learning (DQL) and provides information to it according to the behavior of the VLC system and the objective to maximize the SE. When we use the proposed MDP model in deep Q learning with experienced replay algorithm, we provide the light emitting diode (LED)-based transmitter an autonomy to solve the problem so it can adjust the time-slots of users using the data collected by device in the past. The proposed model includes definitions of the state, actions, and rewards based on the specific characteristics of the problem. Simulations show that the performance of the proposed DQL method can produce results that are competitive to the well-known metaheuristic algorithms, such as Simulated Annealing and Tabu search algorithms.