The 3D surgical tool dataset (3dStool) has been constructed with the aim of assisting the development of computer vision techniques that address the operating room. Note, functions for visualisation, processing, and splitting the dataset can be found in the relevant github repository. Specifically, even though laparoscopic scenes have received a lot of attention in terms of labelled images, surgical tools that are used at initial stages of an operation, such as scalpels and scissors, have not had any such datasets developed. 3dStool includes 5370 images, accompanied by manually drawn polygon labels, as well as information on the 3D pose of these tools in operation. The tools were recorded while operating on a cadaveric knee. A RealSense D415 was used for image collection, while an optical tracker was employed for the purpose of 3D pose recording. Four surgical tools have been collected for now: Scalpel Scissors Forceps Electric Burr An annotation json file (in the format of COCO) exists for the images, containing the masks, boxes, and other relevant information. Furthermore, pose information is provided in two different manners. Firstly, a csv in the following format: CSV Structure Column 1 2 3 4 5 6 7 8 9 Value X (m) Y (m) Z (m) qi qj qk ql Class Image Name Position and orientation are both provided in the coordinate axes of the camera used to obtain the data (Realsense D415, Intel, USA). Pose is provided in the form of quaternions, however it is possible to convert this format into other available notations. The pose data can also be combined with the masks in the form of a final json file, in order to obtain a final COCO-format json with object poses as well. In the data provided, each of the test, train and validation subsets have their own COCO-like json files with the poses fused within, although the "orignal_jsons" only provide the image masks. The files and directories are structured as follows. Note that this example is based on the "train" directory, but a similar structure has been created for the test and val sets: Train manual_json - Contains the json created when manually annotating, the images, therefore no pose data included pose - Contains the CSV file with the poses of the relevant images, explained in the table above pose_json - Contains the fused json that includes both the annotations and the pose data for each image surgical2020 - Contains the images in jpg format