Abstract Convection is usually parameterized in global climate models, and there are often large discrepancies between results obtained with different convection schemes. Conventional methods of comparing convection schemes using observational cases or directly in three‐dimensional (3D) models do not always clearly identify parameterization strengths and weaknesses. In this paper we evaluate the response of parameterizations to various perturbations rather than their behavior under particular strong forcing. We use the linear response function method proposed by Kuang (2010) to compare 12 physical packages in five atmospheric models using single‐column model (SCM) simulations under idealized radiative‐convective equilibrium conditions. The models are forced with anomalous temperature and moisture tendencies. The temperature and moisture departures from equilibrium are compared with published results from a cloud‐resolving model (CRM). Results show that the procedure is capable of isolating the behavior of a convection scheme from other physics schemes. We identify areas of agreement but also substantial differences between convection schemes, some of which can be related to scheme design. Some aspects of the model linear responses are related to their RCE profiles (the relative humidity profile in particular), while others constitute independent diagnostics. All the SCMs show irregularities or discontinuities in behavior that are likely related to threshold‐related mechanisms used in the convection schemes, and which do not appear in the CRM. Our results highlight potential flaws in convection schemes and suggest possible new directions to explore for parameterization evaluation.