Abstract Background Protein-ligand docking has emerged as a particularly important tool in drug design and development, and flexible ligand docking is a widely used method for docking simulations. Many docking software packages can simulate flexible ligand docking, and among them, Autodock is widely used. Focusing on the search algorithm used in Autodock, many new optimization approaches have been proposed over the last few decades. However, despite the large number of alternatives, we are still lacking a search method with high robustness and high performance. Results In this paper, in conjunction with the popular Autodock software, a novel hybrid version of the random drift particle swarm optimization (RDPSO) algorithm, called diversity-guided Lamarckian RDPSO (DGLRDPSO), is proposed to further enhance the performance and robustness of flexible ligand docking. In this algorithm, a novel two-phase diversity control (2PDC) strategy and an efficient local search strategy are used to improve the search ability and robustness of the RDPSO algorithm. By using the PDBbind coreset v.2016 and 24 complexes with apo-structures, the DGLRDPSO algorithm is compared with the Lamarckian genetic algorithm (LGA), Lamarckian particle swarm optimization (LPSO) and Lamarckian random drift particle swarm optimization (LRDPSO). The experimental results show that the 2PDC strategy is able to enhance the robustness and search performance of the proposed algorithm; for test cases with different numbers of torsions, the DGLRDPSO outperforms the LGA and LPSO in finding both low-energy and small-RMSD docking conformations with high robustness in most cases. Conclusion The DGLRDPSO algorithm has good search performance and a high possibility of finding a conformation with both a low binding free energy and a small RMSD. Among all the tested algorithms, DGLRDPSO has the best robustness in solving both holo- and apo-structure docking problems with different numbers of torsions, which indicates that the proposed algorithm is a reliable choice for the flexible ligand docking in Autodock software.