Alzheimer’s disease (AD) is a leading progressive neurodegenerative disease worldwide, and its treatment is a challenging clinical problem. This review was conducted to evaluate the efficacy and safety of herbal medicine for AD treatment. The PubMed, CENTRAL, EMBASE, CNKI, OASIS, KTKP, and CiNii databases were searched until June 2020 for randomized controlled trials (RCTs) on herbal medicine for AD, and a meta-analysis of 57 RCTs was conducted. For cognitive function, herbal medicine significantly improved the Mini-Mental State Examination (MMSE) and AD Assessment Scale-Cognitive Subscale (ADAS-cog) scores compared with conventional medicine. The MMSE scores showed no significant difference between the groups treated with herbal medicine and donepezil; however, herbal medicine significantly lowered the ADAS-cog score. Acori Graminei Rhizoma-containing and Cnidii Rhizoma-containing herbal medicine significantly improved the MMSE and ADAS-cog scores compared with conventional medicine. Ginseng Radix-containing herbal medicine showed a positive, but not statistically significant, tendency toward improving the MMSE score compared with conventional medicine. Herbal medicine with conventional medicine significantly improved the MMSE, ADAS-cog, and Montreal Cognitive Assessment (MoCA) scores compared with conventional medicine, and herbal medicine with donepezil also significantly improved these scores compared with donepezil. Acori Graminei Rhizoma or Cnidii Rhizoma-containing herbal medicine with conventional medicine significantly improved the MMSE and ADAS-cog scores compared with conventional medicine. Ginseng Radix-containing herbal medicine + conventional medicine significantly improved the MMSE score, but not the ADAS-cog score, compared with conventional medicine. For behavioral and psychological symptoms of dementia, the Neuropsychiatry Inventory (NPI) score was not significantly different between herbal and conventional medicines. Herbal medicine with conventional medicine significantly improved the NPI and Behavioral Pathology in Alzheimer’s Disease Rating Scale scores compared with conventional medicine. The NPI score showed no significant difference between the groups treated with herbal medicine and placebo. Furthermore, herbal medicine with conventional medicine significantly lowered plasma amyloid beta levels compared with conventional medicine alone. Herbal medicine, whether used alone or as an adjuvant, may have beneficial effects on AD treatment. However, owing to the methodological limitations and high heterogeneity of the included studies, concrete conclusions cannot be made.