This paper proposes a novel ridge-adding-based approach for handling singularities that are frequently encountered in the powerful SVMpath algorithm. Unlike the existing method that performs linear programming as an additional step to track the optimality condition path in a multidimensional feasible space, our new approach provides a simpler and computationally more efficient implementation, which needs no extra time-consuming procedures other than introducing a random ridge term to each data point. Contrary to the existing ridge-adding method, which fails to avoid singularities as the ridge terms tend to zero, our novel approach, for any small random ridge terms, guarantees the existence of the inverse matrix by ensuring that only one index is added into or removed from the active set. The performance of the proposed algorithm, in terms of both computational complexity and the ability of singularity avoidance, is manifested by rigorous mathematical analyses as well as experimental results.