Abstract Direct imaging of widely separated exoplanets from space will obtain their reflected light spectra and measure their atmospheric properties, and small and temperate planets will be the focus for the next generation of telescopes. In this work, we used our Bayesian retrieval algorithm EXOREL R to determine the constraints on the atmospheric properties of sub-Neptune planets from observations taken with a HabEx-like telescope. Small and temperate planets may have a non-H2-dominated atmosphere, and therefore we introduced the compositional analysis technique in our framework to explore the bulk atmospheric chemistry composition without any prior knowledge about it. We have developed a novel set of prior functions for the compositional analysis free parameters. We compared the performances of the framework with the flat prior and the novel prior and we reported a better performance when using the novel priors set. We found that the retrieval algorithm cannot only identify the dominant gas of the atmosphere but also to constrain other less abundant gases with high statistical confidence without any prior information on the composition. The results presented here demonstrates that reflected light spectroscopy can characterize small exoplanets with diverse atmospheric composition. The Bayesian framework should be applied to design the instrument and the observation plan of exoplanet direct-imaging experiments in the future.