AbstractPurposeArterial spin labeling (ASL) is a widely used contrast‐free MRI method for assessing cerebral blood flow (CBF). Despite the generally adopted ASL acquisition guidelines, there is still wide variability in ASL analysis. We explored this variability through the ISMRM‐OSIPI ASL‐MRI Challenge, aiming to establish best practices for more reproducible ASL analysis.MethodsEight teams analyzed the challenge data, which included a high‐resolution T1‐weighted anatomical image and 10 pseudo‐continuous ASL datasets simulated using a digital reference object to generate ground‐truth CBF values in normal and pathological states. We compared the accuracy of CBF quantification from each team's analysis to the ground truth across all voxels and within predefined brain regions. Reproducibility of CBF across analysis pipelines was assessed using the intra‐class correlation coefficient (ICC), limits of agreement (LOA), and replicability of generating similar CBF estimates from different processing approaches.ResultsAbsolute errors in CBF estimates compared to ground‐truth synthetic data ranged from 18.36 to 48.12 mL/100 g/min. Realistic motion incorporated into three datasets produced the largest absolute error and variability between teams, with the least agreement (ICC and LOA) with ground‐truth results. Fifty percent of the submissions were replicated, and one produced three times larger CBF errors (46.59 mL/100 g/min) compared to submitted results.ConclusionsVariability in CBF measurements, influenced by differences in image processing, especially to compensate for motion, highlights the significance of standardizing ASL analysis workflows. We provide a recommendation for ASL processing based on top‐performing approaches as a step toward ASL standardization.