In 2024 the UK Reproducibility Network (UKRN) initiated a set of pilots involving institutional members and solution providers to establish good practice in institutional monitoring of Open Research through the creation of robust indicators. The Open Research Indicators Pilot was sector led, with institutions and solution providers working together to develop, test, and evaluate prototype machine learning solutions with valid, reliable, and ethical indicators for measuring Open Research. The University of Bristol was the lead for the ‘Openness of Data’ pilot and assessed providers’ data to ascertain the usefulness of machine learning for this purpose. The pilot’s findings highlight the inherent challenges and limitations of monitoring and assessing published datasets for openness within a research landscape that prioritises articles as benchmark outputs; the combination of article primacy and existing publisher and repository systems means datasets can currently only be monitored in Data Availability Statements (DAS). Our analysis of machine learning tools confirmed an uncomfortable truth many in the RDM community suspected; we do not have enough openly available machine actionable metadata for digital tools to reliably and accurately extract DAS, and we are not doing enough at the human interface with researchers to ensure their DAS are easy to understand and describe how their data can be found by others, which impacts measuring openness.