Experiments based on chromosome conformation capture have shown that mammalian genomes are partitioned into topologically associating domains (TADs), within which the chromatin fiber preferentially interacts. TADs may provide three-dimensional scaffolds allowing genes to contact their appropriate distal regulatory DNA sequences (e.g., enhancers) and thus to be properly regulated. Understanding the cell-to-cell and temporal variability of the chromatin fiber within TADs, and what determines them, is thus of great importance to better understand transcriptional regulation. We recently described an equilibrium polymer model that can accurately predict cell-to-cell variation of chromosome conformation within single TADs, from chromosome conformation capture-based data. Here we further analyze the conformational and energetic properties of our model. We show that the chromatin fiber within TADs can easily fluctuate between several conformational states, which are hierarchically organized and are not separated by important free energy barriers, and that this is facilitated by the fact that the chromatin fiber within TADs is close to the onset of the coil-globule transition. We further show that in this dynamic state the properties of the chromatin fiber, and its contact probabilities in particular, are determined in a nontrivial manner not only by site-specific interactions between strongly interacting loci along the fiber, but also by nonlocal correlations between pairs of contacts. Finally, we use live-cell experiments to measure the dynamics of the chromatin fiber in mouse embryonic stem cells, in combination with dynamical simulations, and predict that conformational changes within one TAD are likely to occur on timescales that are much shorter than the duration of one cell cycle. This suggests that genes and their regulatory elements may come together and disassociate several times during a cell cycle. These results have important implications for transcriptional regulation as they support the concept of highly dynamic interactions driven by a complex interplay between site-specific interactions and the intrinsic biophysical properties of the chromatin fiber.