Constitutive equations for the description of optical activity are considered in the scheme of anisotropic nonconducting materials whose response is memory dependent and nonlocal. Attention is then restricted to models containing spatial derivatives up to second order. A dissipation principle is adopted in the form of the Clausius inequality for cycles and, because of nonlocality, the occurrence of an entropy flux is allowed. Thermodynamic restrictions are derived by accounting for the constraints placed by Maxwell's equations and letting the fields be time harmonic. Optically active (chiral) and optically inactive media are examined separately. In the first case thermodynamics is shown to imply the definiteness of the imaginary parts of the permittivity and the permeability along with a bound for the skew-symmetric terms of the real parts. In the second case the occurrence of quadratic terms or higher proves to rule out the possibility of first-order terms.