To find new herbal compounds with an acetylcholinesterase (AChE) inhibitory effect, this study focused on herbal drugs and resins which have been used in Iranian traditional medicine for the treatment of cognitive disorders. Forty drugs were selected from authoritative written documents of Iranian traditional medicine. Each drug was extracted by accelerated solvent extraction using dichloromethane followed by methanol. The 80 extracts were screened for AChE inhibitory activity by a TLC bioautography method. The inhibiting effect of the 32 most active extracts was measured by a microplate colorimetric assay. Due to the best activity, the seeds of Peganum harmala L. were investigated in detail. From the TLC bioautography assay the alkaloids harmaline and harmine were identified as active compounds. This result was confirmed by means of HPLC‐DAD. The IC50 values were 41.2 μg/mL for the methanol extract, 95.5 μg/mL for the dichloromethane extract, 8.4 μg/mL for harmaline and 10.9 μg/mL for harmine. The concentrations of active compounds in the extracts were determined by a fast and precise HPLC method. As the amounts of harmaline and harmine in the extracts were correlated with the IC50 values of the extracts, it can be concluded that these two alkaloids are responsible for the AChE inhibitory activity of P. harmala. Copyright © 2011 John Wiley & Sons, Ltd.