Considering the coexistence of semantic and bit transmissions in future networks, transmission policy design is crucial for heterogeneous semantic and bit communications to improve transmission efficiency. In this paper, we investigate downlink semantic and bit data transmissions from the access point (AP) to several semantic users and a bit user, and we propose a TDMA–NOMA-based transmission scheme to efficiently utilize wireless communication resources. The transmission time and power resource allocation problem is formulated with the aim of maximizing the throughput of the bit user while guaranteeing the semantic demands of the semantic users are met. Due to the time-varying channel conditions and mixed continuous–discrete variables, we propose a parameterized deep Q network (P-DQN)-based algorithm to solve the problem, where an actor network is employed to output continuous parameters, and a deep Q network is used to find the optimal discrete actions. the simulation results show that the proposed P-DQN-based algorithm converges faster than other learning methods. The simulations also demonstrate that the proposed TDMA–NOMA-based transmission scheme can improve the average bit throughput by up to 20% while meeting the semantic demands compared to other multiple access schemes.