This is a pan-tumor immunohistochemistry dataset containing 2 mm2-sized regions of interest (ROIs) from 92 whole slide images stained for cluster of differentiation 3 (CD3, antibody clone SP7). The dataset covers four tumor indications: 32 head and neck squamous cell carcinoma (HNSCC) samples 20 non-small cell lung cancer (NSCLC) samples 20 triple-negative breast cancer (TNBC) samples 20 gastric cancer (GC) samples The samples were digitized at a resolution of 0.23 μm/px (40× objective lens) using the NanoZoomer 2.0-HT scanning system (Hamamatsu, Japan). All samples were prepared and digitized at Merck Healthcare KGaA. We provide bounding box annotations for three cell types: CD3+ immune cells, tumor cells, and other cells. For each tumor indication, five ROIs were manually annotated by three pathologists. For these ROIs, we provide all three annotations and the consensus of all raters. All other ROIs have been semi-automatically annotated using commercial image analysis software.