Let $H$ be an $n$-generated numerical semigroup such that its tangent cone $\operatorname{gr}_\mathfrak{m} K[H]$ is defined by quadratic relations. We show that if $n<5$ then $\operatorname{gr}_\mathfrak{m} K[H]$ is Cohen-Macaulay, and for $n=5$ we explicitly describe the semigroups $H$ such that $\operatorname{gr}_\mathfrak{m} K[H]$ is not Cohen-Macaulay. As an application we show that if the field $K$ is algebraically closed and of characteristic different from two, and $n\leq 5$ then $\operatorname{gr}_\mathfrak{m} K[H]$ is Koszul if and only if (possibly after a change of coordinates) its defining ideal has a quadratic Gröbner basis.