We study a fundamental strategic decision in baseball: when (if at all) to make the ``call to the bullpen'' and pull the starting pitcher. We first use a simple theoretical model to show that at the optimal time to pull the starter, the pitching change should yield a \emph{strict }improvement in current pitching quality (i.e., a strict decrease in runs allowed in the current inning). We then use detailed pitch-level data from the 2008-2017 seasons to estimate the effects of pulling the starter on both runs allowed in the current inning and on win probability. We argue that the pulling starter decision is plausibly ``as good as random'' conditional on the large set of included covariates, but acknowledge the lack of true randomization. We find that the predicted effect of pulling the starter on runs allowed is indeed negative, but the effect on win probability is a precise zero. We then examine how these choices are affected by game situations and recent game events, including a measure of lucky hitting performance, and find only scattered and limited evidence of biases. We interpret the results to imply that call to the bullpen decisions are approximately Bayesian-optimal. However, there was a steady downward trend in the mean inning that starters were pulled over a period of decades prior to our sample time-frame. Thus, even if managers make approximately Bayesian-optimal choices now, this is likely due to not only learning from their own experiences, but also learning from prior generations and the long-term stability of the baseball context.