We have performed microwave–infrared double resonance measurements on OCS in pure He4 and mixed He4/He3 clusters. In contrast to our previous study on OCS in pure He4 cluster, where an asymmetric inhomogeneous linewidth increasing with rotational quantum number j was observed [S. Grebenev, M. Havenith, F. Madeja, J. P. Toennies, and A. F. Vilesov, J. Chem. Phys. 113, 9060 (2000)], we find the lines in the mixed helium cluster to be much narrower, showing a symmetric, probably homogeneous line width. The spectra indicate a significant difference in the rotational relaxation of OCS in mixed clusters as compared to pure He4 cluster. We have applied the model of Lehmann [K. K. Lehmann, Mol. Phys. 97, 645 (1999)], describing the interaction of a molecule with the helium cluster, and modified it in order to describe particle in a box states of OCS in mixed helium clusters. The calculations indicate while in the mixed cluster the OCS is confined to the inner He4 core, the energy of the particle in the box states is increased as compared to the pure He4 cluster. Relaxation mechanisms via these particle in the box states and surface excitations (ripplons) of the cluster are discussed in the text.