X chromosome inactivation (XCI), the silencing of one of the two X chromosomes in XX female cells, equalises the dosage of X-linked genes relative to XY males. The process is mediated by the non-coding RNA X inactive specific transcript (Xist) that binds in cis and propagates along the inactive X chromosome elect, triggering chromosome-wide silencing. The mechanisms by which Xist RNA binds and spreads along the chromosome, and initiates Xist-mediated chromosome silencing remain poorly understood. Accumulating evidence suggests that chromosome and nuclear organisation are important in both processes. Notably, recent studies have identified specific factors, previously shown to be components of the nuclear matrix or scaffold, to play a role both in Xist RNA-binding and in Xist-mediated silencing. In this review we provide a perspective on these studies in the context of previous work on chromosome/nuclear architecture in XCI.