Benchmarking of optimization algorithms is necessary to quantitatively assess the performance of optimizers and to understand their strengths and weaknesses. The Black Box Optimization Benchmarking (BBOB) workshops that took place in 2009, 2010, and 2012 during the Genetic and Evolutionary Computation Conference (GECCO) were set up to benchmark both stochastic and deterministic continuous optimization algorithms. For this purpose, a thorough experimental setting, a set of test functions, and a visualization tool were designed and provided. They are based on the idea that (i) test functions should be representative of typical known difficulties, scalable with dimension, and not too easy to solve, yet comprehensible; and (ii) performance measures should be quantitative. A tool for acquiring and postprocessing data was provided.