After the fire that destroyed most of the Notre-Dame de Paris cathedral's roof and vaults, scientists gathered in an effort to help the restoration process of the cathedral. Several digital methods and heterogeneous data acquisitions were introduced in the process, including many images and annotations. Part of this data focuses on stone degradation phenomena, a crucial element when evaluating the damages caused by the fire and the state of the cathedral before the restoration started. In this paper, we present the first implementation of a dataset creation pipeline with the aim of training AI models to automatically detect and segment stone alteration patterns in images taken in the context of the restoration of Cultural Heritage buildings. Our resulting dataset will be improved in a near future with more data, while conforming with the ambition to provide our experts and researchers with reliable, structured data.