Abstract Flexible sheets are used in many industrial applications. During their manufacturing processes, the coupling of the sheet motion and airflow causes flutter, which can result in severe quality defects. To prevent these defects, a detailed understanding of the fluttering mechanisms and characteristics is essential. In this study, wind-tunnel experiments and three-dimensional nonlinear flutter analysis are carried out on a cantilevered rectangular sheet in uniform flow. The post-critical behavior of the sheets (flutter amplitudes, frequencies and modes) is examined in detail. The influence of structural damping and fluid friction on the post-critical behavior is carefully investigated for various mass ratios through the nonlinear analysis. Finally, the work done by the fluid force acting on the sheet surface is examined through nonlinear analysis to understand the sustaining mechanism of limit cycle oscillation (LCO). It is found that the competition between the positive and negative work done by the fluid force contributes considerably to the sustaining mechanism of the LCO.