This dataset consists of resource metadata harvested from the APIs of hundreds of government and research data portals from all over the world. This dataset was harvested between the 13th and 15th of September 2018. The metadata harvested from these portals was translated to a single metadata format (see metadata_format.odt). An overview of all harvested domains is given in portal_list.txt. The harvested data is divided into five gzipped json-lines files, based on the ‘type’ of the resource that is derived from the data of the APIs: dataset_metadata.jsonl.gz: Resources classified as a Dataset, or subsets of dataset (e.g. Dataset:Image and Dataset:Audio) [6 246 250 resources] document_metadata.jsonl.gz: Resources classified as a Document, or subset of document (e.g. Document:Paper:Conference and Document:Book) [15 626 541 resources] software_metadata.jsonl.gz: Resources classified as Sofware (including Software:Model) [42 036 resources] service_metadata.jsonl.gz: Resources classified as a service (e.g. WMS, APIs) [1257 resources] other_metadata.jsonl.gz: Resources of which the ‘type’ could not be determined from the data the API returned. This set still contains many datasets [1 502 979 resources]

The data is provided as Gzipped JSON lines files, further defined here: http://jsonlines.org/