Terahertz band (0.1-10 THz) communication is envisioned as a key technology to support ultra-broadband wireless systems for beyond 5G. For realization of efficient wireless communication networks in the THz band, it is imperative to develop channel models that can accurately and efficiently characterize the THz spectrum peculiarities. This article provides an in-depth view of channel modeling in the THz band, based on the deterministic, statistical, and hybrid methods. The state-of-the-art THz channel models in single-antenna and ultra-massive MIMO systems are extensively reviewed, respectively. Furthermore, the open challenges and potential research directions are highlighted regarding THz propagation modeling. Associated with the channel models, key physical parameters of the THz channel and their implications for wireless communication design are analyzed. The provided analysis lays the foundation for reliable and efficient ultra-broadband wireless communications in the THz band.