To prevent the population from getting stuck in local areas and then missing the constrained Pareto front fragments in dealing with constrained multi-objective optimization problems (CMOPs), it is important to guide the population to evenly explore the promising areas that are not dominated by all examined feasible solutions. To this end, we first introduce a cost value based distance into the objective space, and then use this distance and the constraints to define an indicator to evaluate the contribution of each individual to exploring the promising areas. Theoretical studies show that the proposed indicator can effectively guide population to focus on exploring the promising areas without crowding in local areas. Accordingly, we propose a new constraint handling technique (CHT) based on this indicator. To further improve the diversity of population in the promising areas, the proposed indicator-based CHT divides the promising areas into multiple subregions, and then gives priority to removing the individuals with the worst fitness values in the densest subregions. We embed the indicator-based CHT in evolutionary algorithm and propose an indicator-based constrained multi-objective algorithm for solving CMOPs. Numerical experiments on several benchmark suites show the effectiveness of the proposed algorithm. Compared with six state-of-the-art constrained evolutionary multi-objective optimization algorithms, the proposed algorithm performs better in dealing with different types of CMOPs, especially in those problems that the individuals are easy to appear in the local infeasible areas that dominate the constrained Pareto front fragments.