Abstract This paper proposes a method for the automatic construction of a dataset with annotation data for object detection. The accuracy of the object detection method depends on the dataset in general. The dataset for object detection needs many images with annotation data. Obtaining image data by manual operation takes a lot of costs. It also costs much that annotation data are made by manual annotation software. This paper tries to solve these problems to construct the image dataset for the object detection automatically. The proposed method uses a method to collect the image data automatically among images on the Web using Web image mining. A new method for making annotation data is also proposed. The Mask R-CNN is used for automatical annotation. The proposed approach constructs a dataset automatically without almost manual operation. It is confirmed that the proposed approach performs the automatic construction of the dataset with high accuracy.