We introduce the first video-based retinal vessel dataset (RVD), a collection of 635 smartphone-based videos with detailed vessel annotation. All captured videos have a frame rate of 25 frames per second, with the duration varying between 2 to 30 seconds. The total number of frames in our dataset is over 130,000. These videos are recorded from four clinics, including patients from 50 to 75 years old. More specifically, 264 males and 151 females are included in the collection process. The annotations provided in our dataset span two dimensions: spatial and temporal. In the spatial dimension, we offer three distinct levels of annotations: binary vessel masks, general vein-artery masks, and fine-grained vein-artery masks. Each kind of annotation is tailored to specific clinical purposes. In the temporal dimension, we focus on the optic disk regions of videos where the retinal vessel fluctuation normally occurs. We select and annotate frames with the maximal and minimal pulse widths as well as label the existence of spontaneous retinal venous pulsations (SVP). More detailed information can also be found on our website: https://uq-cvlab.github.io/Retinal-Video-Dataset/ If you find our RVD dataset is useful in your research, please consider cite: @article{MD2023RVD, title={RVD: A Handheld Device-Based Fundus Video Dataset for Retinal Vessel Segmentation}, author={MD WAHIDUZZAMAN KHAN, Hongwei Sheng, Hu Zhang, Heming Du, Sen Wang, Minas Theodore Coroneo, Farshid Hajati, Sahar Shariflou, Michael Kalloniatis, Jack Phu, Ashish Agar, Zi Huang, Mojtaba Golzan, Xin Yu}, journal={arXiv preprint arXiv:2307.06577}, year={2023} }