We analyse the performance of direct solvers when applied to a system of linear equations arising from an $h$-adapted, $C^0$ finite element space. Theoretical estimates are derived for typical $h$-refinement patterns arising as a result of a point, edge, or face singularity as well as boundary layers. They are based on the elimination trees constructed specifically for the considered grids. Theoretical estimates are compared with experiments performed with MUMPS using the nested-dissection algorithm for construction of the elimination tree from METIS library. The numerical experiments provide the same performance for the cases where our trees are identical with those constructed by the nested-dissection algorithm, and worse performance for some cases where our trees are different. We also present numerical experiments for the cases with mixed singularities, where how to construct optimal elimination trees is unknown. In all analysed cases, the use of $h$-adaptive grids significantly reduces the cost of the direct solver algorithm $per$ $unknown$ as compared to uniform grids. The theoretical estimates predict and the experimental data confirm that the computational complexity is linear for various refinement patterns. In most cases, the cost of the direct solver $per$ $unknown$ is lower when employing anisotropic refinements as opposed to isotropic ones.