Bott-Samelson varieties are a twisted product of $\mathbb{C}\mathbb{P}^1$'s with a map into $G/B$. These varieties are mostly studied in the case in which the map into $G/B$ is birational to the image; however in this paper we study a fiber of this map when it is not birational. We prove that in some cases the general fiber, which we christen a brick manifold, is a toric variety. In order to do so we use the moment map of a Bott-Samelson variety to translate this problem into one in terms of the "subword complexes" of Knutson and Miller. Pilaud and Stump realized certain subword complexes as the dual of the boundary of a polytope which generalizes the brick polytope defined by Pilaud and Santos. For a nice family of words, the brick polytope is the generalized associahedron realized by Hohlweg, Lange and Thomas. These stories connect in a nice way: we show that the moment polytope of the brick manifold is the brick polytope. In particular, we give a nice description of the toric variety of the associahedron. We give each brick manifold a stratification dual to the subword complex. In addition, we relate brick manifolds to Brion's resolutions of Richardon varieties.