This dataset is presented in the paper A large brain organoid dataset reveals extensive improvement potential of state-of-the-art analysis pipelines. This dataset encompasses two sources of data: A comma-separated values (‘CSV’) file. This file serves as a key to our dataset with one image per row. Each image is represented by its image identifier (‘img_id’) with the format [img_nr]_d[imaging_day]_[clone]. For each image, the CSV also specifies the organoid size for convenience. Alternatively, the organoid size can be calculated using the ground truth organoid segmentation (org_segGT). For each row of the CSV, we provide the image (.jpg file) and org_segGT (.npy file). org_segGT is a manually created binary 2D NumPy array with the same size as the image (1024 x 768). A value of 1 in org_segGT at position (x, y) means that the same position (x, y) in the corresponding image is covered by the organoid. The image file and the org_segGT file have the following format: [img_id].jpg and [img_id]_org-mask.npy. For segmentation and growth monitoring using this dataset, please see https://github.com/deiluca/robust_monitoring_organoid_growth.