We evaluate performance profiles as a method for comparing preconditioners for iterative solvers by using them to address three questions that have previously been asked about incomplete LU preconditioners. For example, we use performance profiles to quantify the observation that if a system can be solved by a preconditioned iterative solver, then that solver is likely to use less space, and not much more time, than a direct solver. In contrast, we also observe that performance profiles are difficult to use for choosing specific parameter values. We end by discussing the role performance profiles might eventually play in helping users choose a preconditioner.