Distributed optimization strategies are promising solutions for direct load control (DLC), since the centralized ones have privacy and heavy communication issues. However, conventional synchronous distributed methods suffer from communication delay or failures. In this paper, we present an asynchronous distributed approach based on a Fast Alternative Direction Method of Multipliers (ADMM) in which a synthetic asynchronous strategy is adopted to tackle with the time-delay into the upload and download link. To accelerate the developed algorithm’s convergence further, methods including adaptive updating of penalty parameter, warm start and over-relaxation are introduced. Since the residential loads are connected on low voltage level networks, three-phase unbalanced distribution network is formulated in the DLC model. The objective considers the operational cost of LSE and the utility function of consumers. Numerical tests on IEEE 33-bus and IEEE 123-bus systems show that the proposed method can realize reliable and fast convergence both with and without communication delay, attaining the optimal solution consistent with the centralized algorithm simultaneously, and requires much less computing time compared with the synchronous algorithm.