Blurring an image with a Gaussian of width σ and considering σ as an extra dimension, extends the image to an Gaussian scale space ($\mathcal{GSS}$) image. In this $\mathcal{GSS}$-image the iso-intensity manifolds behave in an nicely pre-determined manner. As a result of that, the $\mathcal{GSS}$-image directly generates a hierarchy in the form of a binary ordered rooted tree, that can be used for segmentation, indexing, recognition and retrieval. Understanding the geometry of the manifolds allows fast methods to derive the hierarchy. In this paper we discuss the relevant geometric properties of $\mathcal{GSS}$ images, as well as their implications for algorithms used for the tree extraction. Examples show the applicability and increased speed of the proposed method compared to traditional ones.