This dataset contains multimodal images captured in a real waste processing facility specialized in plastics, cartons, and cans. The data was collected with a true-to-life prototype of the conveyor belt installed on the waste separation line, closely mimicking the actual installation. The setup has two synchronized cameras: a line-scan RGB camera (Teledyne DALSA Linea) and line-scan hyperspectral sensor (Specim FX17) that captures 224 contiguous spectral bands in a range from 900 to 1700 nm. The images are annotated for semantic segmentation with categories based on the requirements of the facility. Each one represents elements that commonly cause operational problems in recycling lines and impact the efficiency of the sorting process. Among these problems, machinery jams pose significant issues because they can cause complete stoppages in the process until the obstructing object is removed. The categories include: film and basket, large objects that can clog the conveyor belts as they are not easily breakable; video tape and filament, representing long objects prone to entangling with mechanical parts and requiring manual intervention; trash bag, which encompasses closed bags containing waste that need to be mechanically opened for further processing; and cardboard, paper objects whose recovery adds value by sending them to another recycling process. Our project page can be found at https://sites.google.com/unizar.es/spectralwaste.