Adenocarcinoma of the ampulla of Vater (AOV) is classified into intestinal type (IT) and pancreatobiliary type (PB); however, the immunological properties of these subtypes remain to be characterized. Here, we evaluated the clinical implications of PD-L1 expression and CD8+ T lymphocyte density in adenocarcinomas of the AOV and their potential association with Yes-associated protein (YAP). We analyzed 123 adenocarcinoma-of-the-AOV patients who underwent surgical resection, and tumors were classified into IT or PB type. Tumor or inflammatory cell PD-L1 expression, CD8+ T lymphocyte density in the cancer cell nest (intratumoral) or in the adjacent stroma, and YAP localization and intensity were analyzed using immunohistochemical staining. PB-type tumors showed higher tumoral PD-L1 expression than IT-type tumors, and tumoral PD-L1 expression was associated with a shorter disease-free survival (DFS) [hazard ratio (HR), 1.77; p = 0.045] and overall survival (OS) (HR 1.99; p = 0.030). Intratumoral CD8+ T lymphocyte density was higher in IT type than in PB type and was associated with a favorable DFS (HR 0.47; p = 0.022). The nuclear staining pattern of YAP in tumor cells, compared to non-nuclear staining patterns, was more frequently associated with PB type and increased tumoral PD-L1 expression. Nuclear YAP staining was a significant prognostic factor for OS (HR 2.21; p = 0.022). These results show that the two subtypes of adenocarcinoma of the AOV exhibit significant differences in tumoral PD-L1 expression and intratumoral CD8+ T lymphocyte density, which might contribute to their distinct clinical features.