We investigated the effect of sitosterol on hepatic sterol 27-hydroxylase activities in subjects with sitosterolemia, a recessive inherited disease associated with accelerated atherosclerosis and increased levels of sitosterol and other plant sterols and stanols in tissues. Hepatic activities of mitochondrial sterol 27-hydroxylase, which catalyzes the first step in the conversion of cholesterol to bile acids via the acidic bile acid synthetic pathway, were measured in liver tissues and related to hepatic microsomal cholesterol 7 alpha-hydroxylase, which controls the rate of bile acid synthesis via the neutral synthetic pathway. These measurements of cholesterol catabolism were correlated to sterol concentrations and composition in plasma and liver. Sterol 27-hydroxylase activities in liver mitochondria of three homozygous sitosterolemic subjects were 68% lower than in 10 control subjects (p < .05) and were associated with increased levels of plant sterols in both plasma and liver (13% and 16% of total sterols, respectively, compared to trace amounts in controls). Analysis of Lineweaver-Burk double reciprocal plots of sterol 27-hydroxylase activities in control human liver specimens (where mitochondrial sterol 27-hydroxylase activities were measured with increasing concentrations of the cholesterol substrate, in the absence and presence of 100 microM and 300 microM sitosterol) revealed that sitosterol inhibited mitochondrial sterol 27-hydroxylase activity up to 50% by a competitive mechanism. In sitosterolemic subjects, competitive inhibition of hepatic sterol 27-hydroxylase activity by sitosterol was associated with competitively inhibited microsomal cholesterol 7 alpha-hydroxylase activity (averages from 4 sitosterolemic homozygotes and 14 controls were 12.4 +/- 1.9 and 23.6 +/- 2.5 pmol/mg/min, respectively). Furthermore, decreased cholesterol catabolism in sitosterolemic subjects was associated with significantly elevated plasma cholesterol concentrations (232 +/- 17 mg/dl, as compared to 180 +/- 13 mg/dl in controls) but with no change in hepatic cholesterol concentrations. In an animal model (rats infused intravenously with sitosterol-containing liposomes that increased sitosterol in the liver and plasma to levels similar to those found in sitosterolemic subjects), hepatic mitochondrial sterol 27-hydroxylase and microsomal cholesterol 7 alpha-hydroxylase activities also decreased significantly and were coupled to markedly elevated plasma sterol concentrations (120.7 +/- 12.5 mg/dl, as compared to 59.2 +/- 6.3 mg/dl in control animals; p < .05) but to no change in hepatic cholesterol concentrations. Thus, decreased cholesterol catabolism due to competitive inhibition of both microsomal cholesterol 7 alpha-hydroxylase and mitochondrial sterol 27-hydroxylase by elevated hepatic sitosterol concentrations contributes to hypercholesterolemia and increased risk of atherosclerosis in sitosterolemia.