Background This study examines the practical implementation of the Agreement on Reforming Research Assessment within the National Research Council of Italy by analysing the 2023 competitive procedures for the career progression of researchers and technologists. As the institution formally adopted the reform to shift towards a more responsible and qualitative evaluation framework, this research investigates the extent to which these high-level commitments were applied within the calls by the individual evaluation committees. Methods A systematic qualitative and quantitative analysis was conducted on 4,090 evaluation criteria established by 90 independent evaluation committees. The criteria were examined to assess their alignment with the four core commitments of the Agreement and the inclusion of Open Science principles. Results Results indicate a significant gap between institutional policy and actual practice: most committees do not fully recognise the diversity of research contributions, often undervaluing or excluding non-traditional outputs, such as teaching and evaluation activities, or public and policy engagement. Quantitative and mechanical metrics continue to be widely used in researcher evaluations, with journal-based indicators often used to assess research quality. Regarding Open Science practices, they appeared only sporadically in the assessed criteria. Conclusions These findings suggest that traditional evaluation practices persist at the committee level, despite formal institutional reform. The study highlights that high-level policy adoption alone is insufficient without robust guidance to operationalise reform principles during the assessment process. This research provides insights into the challenges of transitioning from policy to practice in research assessment within a large, multidisciplinary Research Performing Organisation.