Abstract Differential evolution (DE) has been a simple yet effective algorithm for global optimization problems. The performance of DE highly depends on its operators and parameter settings. In the last couple of decades, many advanced variants of DE have been proposed by modifying the operators and introducing new parameter tuning methods. However, the majority of the works on advanced DE have been concentrated upon the mutation and crossover operators. The initialization and selection operators are less explored in the literature. In this work, we implement the orthogonal array-based initialization of the population and propose a neighborhood search strategy to construct the initial population for the DE-based algorithms. We also introduce a conservative selection scheme to improve the performance of the algorithm. We analyze the influence of the proposed initialization and selection schemes on several variants of DE. Results suggest that the proposed methods highly improve the performance of DE algorithm and its variants. Furthermore, we introduce an ensemble strategy for parameter adaptation techniques in DE. Incorporating all the proposed initialization, selection, and parameter adaptation strategies, we develop a new variant of DE, named OLSHADE-CS. The performance of OLSHADE-CS is found to be highly competitive and significantly better in many cases when compared with the performance of the state-of-the-art algorithms on CEC benchmark problems.