Elasticity and recovery are important for clothing comfort, especially in the manufacture of apparel and sportswear. Recently, yarns containing PBT (polybutylene terephthalate), which are able to develop good elastic properties with high recovery after a finishing process (e.g., thermal treatment), have been used for this purpose. The aim of this work is to give a comprehensive overview of the use of PBT yarns in woven structure, with the aim of improving the elastic properties of cotton-like fabrics. The experimental part was divided into three main sequences to investigate the fabric properties (physical, elastic, UPF, comfort) influenced by (1) PBT-containing yarn structure, (2) weave and fabric structure (basic weaves and complex weaves) with PBT in weft direction, and (3) processing sequence—thermal treatment of PBT yarns or fabrics after weaving. According to the results, PBT-containing yarns have great potential for the production of lightweight elastic fabrics. The advantages of improving the elastic properties of fabrics by incorporating a relatively small amount of PBT yarns into the fabric only in certain areas, thereby minimally affecting the production costs, are demonstrated by a product with partially elastic areas obtained after thermal treatment.