We report medium resolution VLT ISAAC K-band spectroscopy of the nuclei of seven ultraluminous infrared galaxies. After accounting for stellar absorption features, we have detected several molecular hydrogen (H_2) v=1-0, 2-1, and 3-2 vibrational emission lines, as well as the HI Br��and HeI 2^1P-2^1S recombination lines. The relative H_2 line intensities show little variation between the objects, suggesting that the H_2 excitation mechanisms in the nuclei are similar in all the objects. The 1-0 emissions appear thermalised at temperatures T\sim1000K. However, the 2-1 and 3-2 emissions show evidence of being radiatively excited by far-ultraviolet (FUV) photons, suggesting that the H_2 excitation in the ULIRGs may arise in dense photon dominated regions (PDRs). We show that the line ratios in the nuclei are consistent with PDRs with cloud densities between 10^4 to 10^5cm^{-3}, exposed to far ultraviolet (FUV) radiation fields at least 10^3 times more intense than the ambient FUV intensity in the local interstellar medium. We have constructed starburst models for the ULIRGs based on their H_2 properties, as well as on the intensities of the recombination lines. Our models provide a consistent picture of young 1-5Myr star clusters surrounded by relatively dense PDRs which are irradiated by intense FUV fluxes. Comparison to the inner few hundred parsecs of the Milky Way indicates that the star formation efficiency in ULIRGs is 10--100 times higher than in the Galactic Center.

accepted by ApJ (32 pages including figures)