AbstractLocus heterogeneity is a major confounding factor in linkage analysis. When no prior knowledge of linkage exists, and one aims to detect linkage and heterogeneity simultaneously, classical distribution theory of log‐likelihood ratios does not hold. Despite some theoretical work on this problem, no generally accepted practical guidelines exist. Nor has anyone rigorously examined the combined effect of testing for linkage and heterogeneity and simultaneously maximizing over two genetic models (dominant, recessive). The effect of linkage phase represents another uninvestigated issue. Using computer simulation, we investigated type I error (P value) of the “admixture” heterogeneity LOD (HLOD) score, i.e., the LOD score maximized over both recombination fraction θ and admixture parameter α and we compared this with the P values when one maximizes only with respect to θ (i.e., the standard LOD score). We generated datasets of phase‐known and ‐unknown nuclear families, sizes k = 2, 4, and 6 children, under fully penetrant autosomal dominant inheritance. We analyzed these datasets (1) assuming a single genetic model, and maximizing the HLOD over θ and α; and (2) maximizing the HLOD additionally over two dominance models (dominant vs. recessive), then subtracting a 0.3 correction. For both (1) and (2), P values increased with family size k; rose less for phase‐unknown families than for phase‐known ones, with the former approaching the latter as k increased; and did not exceed the one‐sided mixture distribution ξ = (½) χ + (½) χ. Thus, maximizing the HLOD over θ and α appears to add considerably less than an additional degree of freedom to the associated χ distribution. We conclude with practical guidelines for linkage investigators. Genet. Epidemiol. 22:156–169, 2002. © 2002 Wiley‐Liss, Inc.