Computer programs for simulation and auralization of sound fields in rooms and of sound transmission in buildings became standard tools for architectural design and consulting. Another field of rapid progress is Virtual Reality—the combination of real-time signal processing with user interaction and multimodal human-machine interfaces. In this contribution, the development of simulation tools in architectural acoustics and further work aiming at real-time Acoustic Virtual Reality systems are reviewed and discussed with emphasis on the challenges and interdisciplinary solutions involving architectural acoustics as well as signal processing and psychoacoustics.