Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in designing benchmarks for generalist robotic manipulation policies for the goal of sim-to-real policy transfer. We propose 1) utilizing high visual-fidelity simulation for improved sim-to-real transfer, 2) evaluating policies by systematically increasing task complexity and scenariResearch goal: How robust are DPPO-trained diffusion-based policies to distribution shifts in real-world robotic manipulation scenarios, as evaluated by performance metrics on out-of-distribution test sets or sim-to-real transfer benchmarks like RoboReal or MIT Adaptation?Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.9/10.