The aim of this study was to identify bioactive compounds from leaves of the invasive plant Xanthium spinosum and assess their phytotoxic activity. Activity-guided fractionation led to the isolation of 6 bioactive compounds: xanthatin (1), 1α,5α-epoxyxanthatin (2), 4-epiisoxanthanol (3), 4-epixanthanol (4), loliolide (5) and dehydrovomifoliol (6). Of them, compounds 2–6 were isolated from the X. spinosum for the first time. The structures of 1–6 were elucidated on the basis of extensive NMR studies and ESI-MS measurements as well as comparison with literature data. All of compounds were evaluated for their phytotoxic activity. Among them, compounds 1–4 exhibited stronger activity on 2 receiver plants compared with the other 2 compounds, with xanthatin (1) being the most potent compound, which suppressed root growth of the dicot plant Amaranthus retroflexus by 32.5%, 39.4%, 84.7% when treated xanthatin (1) at 5, 20, and 100 µg/mL, while for the monocot plant, root growth was inhibited by 14.7%, 28.0%, and 40.0%, respectively. Seedling growth was nearly completely inhibited when the concentration of xanthanolides increased to 500 µg/mL, whereas there was still some seedling growth when loliolide (5) and dehydrovomifoliol (6) were applied at the same concentration. Dehydrovomifoliol (6) did not negatively affect seedling growth of P. annua at all tested concentrations, and root length was still 42.0% of the control when the highest concentration 500 µg/mL was used. This is the first report of the phytotoxicity of 1α,5α-epoxyxanthatin (2), 4-epiisxanthanol (3) and 4-epixanthanol (4). These compounds have the potential to be utilized as natural herbicides, especially 4-epiisoxanthanol (3), which exhibited significant selective activity between the dicot and monocot plants. On the other hand, whether these bioactive substances serve as allelochemicals to facilitate the invasion success of X. spinosum needs to be further studied.