We consider the optimal control problem for networks subjected to time-varying channels, reconfiguration delays, and interference constraints. We show that the simultaneous presence of time-varying channels and reconfiguration delays significantly reduces the system stability region and changes the structure of optimal policies. We first consider memoryless channel processes and characterize the stability region in closed form. We prove that a frame-based Max-Weight scheduling algorithm that sets frame durations dynamically, as a function of the current queue lengths and average channel gains, is throughput-optimal. Next, we consider arbitrary Markov-modulated channel processes and show that memory in the channel processes can be exploited to improve the stability region. We develop a novel approach to characterizing the stability region of such systems using state-action frequencies, which are stationary solutions to a Markov Decision Process (MDP) formulation. Moreover, we develop a dynamic control policy using the state-action frequencies and variable frames whose lengths are functions of queue sizes and show that it is throughput-optimal. The frame-based dynamic control (FBDC) policy is applicable to a broad class of network control systems, with or without reconfiguration delays, and provides a new framework for developing throughput-optimal network control policies using state-action frequencies. Finally, we propose Myopic policies that are easy to implement and have better delay properties as compared to the FBDC policy.