Customizing Segment Anything Model (SAM) has recently attracted considerable attention in remote sensing domains. This study explores the performance of SAM-Adapter in refugee-dwelling extraction in three different refugee camps from high-resolution satellite images. The findings indicate that with scarce sample data, SAM-Adapter marginally outperforms other semantic segmentation models. This underscores SAM's promising potential for building extraction tasks when data is limited.