Classical heritability models for family data split the phenotype variance into genetic and environmental components. For instance, the ACE model in twin studies assumes the phenotype variance decomposes as a2 + c2 + e2, representing (additive) genetic effects, common (shared) environment, and residual environment, respectively. However, for some phenotypes it is biologically plausible that the genetic and environmental components may vary over the range of the phenotype. For instance, very large or small values of the phenotype may be caused by “sporadic” environmental factors, whereas the mid‐range phenotype variation may be more under the control of common genetic factors. This article introduces a “local” measure of heritability, where the genetic and environmental components are allowed to depend on the value of the phenotype itself. Our starting point is a general formula for local correlation between two random variables. For estimation purposes, we use a multivariate Gaussian mixture, which is able to capture nonlinear dependence and respects certain distributional constraints. We derive an analytical expression for the associated correlation curve, and show how to decompose the correlation curve into genetic and environmental parts, for instance, a2(y) + c2(y) + e2(y) for the ACE model, where we estimate the components as functions of the phenotype y. Furthermore, our model allows switching, for instance, from the ACE model to the ADE model within the range of the same phenotype. When applied to birth weight (BW) data on Norwegian mother‐father‐child trios, we conclude from the model that low and high BW are less heritable traits than medium BW. We also demonstrate switching between the ACE and ADE model when studying body mass index in adult monozygotic and dizygotic twins.