New methods for storing large volumes of data, such as cloud systems and object storage present opportunities: • Improved access and user management over POSIX • Parallel reads / writes • Read / write from anywhere (no need to mount disks) and challenges: • Interface and access methods for users are changed • Exploit parallel and remote access nature of stores with S3 HTTP API • While maintaining a familiar and consistent API for the user • S3-netCDF-python: • Library to read/write netCDF 3 and netCDF 4 via a S3 HTTP API • Files can be streamed directly to memory or cached to disk • Files can be split into smaller objects • Multiple objects can be streamed in parallel

Previously curated at: http://cedadocs.ceda.ac.uk/1362/ Event type: conference. This item was not refereed before the publication Main files in this record: nrmassey_ncas2018_poster.pdf Item originally deposited with Centre for Environmental Data Analysis (CEDA) document repository by Dr Neil Massey. Transferred to CEDA document repository community on Zenodo on 2022-11-24