High expectation is given to the device-to-device (D2D) communication to improve the system capacity in cellular networks. But, the interference among specturm-sharing links is a major obstacle to fulfill the potential of D2D communication. To the end, we focus on joint optimization of mode selection, channel allocation, and power control in a direct and relay D2D communication coexisted scenario. The paper proposed a novel resource allocation scheme aiming at minimizing overall energy consumption, subject to the quality-of-service (QoS) requirements of cellular users (CUs) and D2D links. In order to solve the formulated non-convex combinatorial optimization problem, a three-stage scheme is proposed. Specifically, the joint mode selection and relay selection (JMSRS) algorithm is proposed to select communication modes for D2D links firstly. Then, the relay enhanced channel assignment (RCA) algorithm is used to determine the set of reused channels of each D2D link. Finally, the relay-enhanced power control (RPC) is devised to coordinate the transmit power of the BS, relays, and D2D transmitters to alleviate the interference. The simulation results demonstrate that the proposed scheme can significantly improve the system performance in terms of data rate and energy consumption, while guaranteeing diverse QoS requirements of the CUs and D2D links.