With the advancement of small watershed governance in agricultural production process, soil toxic element pollution issue in watersheds constitutes a recent research hot spot. The Ashi River watershed is an agriculture-dominated small watershed which is exposed to toxic element sources, posing high risk of toxic element pollution to the planting areas. In this study, collection of soil samples was carried out along the periphery of the river network, and the soil physicochemical parameters and toxic elements (As, Cd, Cr, Cu, Pb, and Zn) were analyzed. The results showed that: (1) The geo-accumulation index (Igeo) and potential ecological risk index were used to evaluate the pollution degree, and the contents of As, Cd, and Zn in some sampling sites exceeded risk screening values. Moreover, soils closer to mining sources were found to be more polluted; (2) Redundancy analysis confirmed the contribution rate relationship between environmental factors and toxic elements. C/N ratio, total carbon (C), and total potassium (K) exhibited significant relationships with toxic elements (P < 0.01 or P < 0.05), respectively. Moreover, geographic locations (longitude, latitude, and elevation) showed significant impacts on toxic element contents (except for Cu); (3) The apportionment of toxic element pollution sources by using principal component analysis showed that Pb, Zn, Cu, and Cd were mainly related to mining activities, while As was closely related to insecticide and herbicide, and Cr was mainly related to soil parent material and electroplating factory; (4) Through the integrated resistance base surface and toxic element sources combined with minimum cumulative resistance model, the toxic element risk areas were identified. The middle reaches corresponded to the extremely high risk zone, which undeniably requires the strengthening of the environmental management.