Interpreting probability in the context of individual patients remains conceptually difficult. The common interpretation is that a 5% probability means 5 out of 100 similar patients will experience the event. This is suspect to many: what does this say about a unique individual, here and now, given that no two patients are truly identical? Based on an exploration of philosophical and statistical perspectives, we address two misconceptions: that individual probabilities can be objectively true, and, inversely, that they are necessarily arbitrary and therefore meaningless. We argue that good predictions are rational, conditional estimates of uncertain patient outcomes, based on a specified reference class. As such, estimated predictions must be critically evaluated through clinical expertise. Clinical judgments, in turn, must be constrained by empirical data. A clearer common understanding of the limitations of modeling and estimating probabilities influences how we use AI, structure clinical reasoning, and inform patients about uncertain outcomes.