In this paper, the problem of motion planning of an autonomous agent in an uncertain environment is considered. The state of the augmented system is defined as the ordered pair consisting of the state of the agent and the state of the environment. At the higher level of the hierarchy, the state-space of the agent is reduced to a set of "landmarks" through the use of suitably defined control policies at the lower level, called "options". Assuming that the environment is stationary, it (along with its associated uncertainties) is abstracted into a lower dimensional global variable through a suitably designed feature-map. This results in a drastic reduction of the computational complexity of the planning algorithms. Error bounds are obtained on the deviation of the approximate policies from the optimal policies, and the methodology is applied to an unmanned ground vehicle navigating a cluttered and uncertain urban environment.