Abstract Solid-state dye-doped polymer is an attractive alternative to the conventional liquid dye solution. In this paper the spectral characteristics and the nonlinear optical properties of the dye malachite green in polymethylmethacrylate is studied. The spectral characteristics are studied by recording steady state absorption and fluorescence spectra. Thermally induced nonlinearity of malachite green dye in n-butyl Acetate is studied using cw He–Ne laser at 632.8 nm as source of excitation, both in solution and solid film. The property of dye in solid matrix is compared with that in liquid medium. The optical response is characterized by measuring the intensity dependent refractive index (n2) of the medium, using z-scan technique. The origin of optical nonlinearity in this dye may be attributed due to laser-heating induced nonlinear effect. The dye exhibited negative (defocusing) nonlinearity and it can be a promising material for optical limiting applications.