SummaryMobile broadband interactive satellite communication system is of great interest in both academic and industrial communities. However, the conventional strict‐layered protocol stack architecture and the standard TCP version perform poorly over satellite link. In this paper, we propose a comprehensive cross‐layer Transmission Control Protocol (TCP) optimization architecture while considering the main factors that affect the TCP performance. In our proposed architecture, we adopt two TCP split connection performance enhancing proxies to isolate the satellite link from the terrestrial part of the broadband satellite communication system. Then, based on the proposed cross‐layer architecture, we present an analytical model for the TCP throughput by taking the modulation and coding (ModCod) mode and the allocated bandwidth into account. In addition, we put forward a TCP‐driven bandwidth sharing and ModCod mode optimization algorithm to maximize the TCP throughput in satellite link. Extensive simulation results illustrate that our proposed comprehensive cross‐layer TCP optimization approach is able to improve the TCP throughput significantly. Copyright © 2014 John Wiley & Sons, Ltd.