The diversity of agricultural genetic resources is decreasing over the years and sites across the world. With the objectives of determining the drivers and their impact on agrobiodiversity particularly in Nepal, different methods e.g., focus group discussion, transact walk, key informant survey, and literature survey were used. Among the 16 broad categories of drivers, the most 5 important drivers in terms of negative impact on agrobiodiversity were the: 1) advancements and innovations in science and technology, 2) changes in land and water use and management, 3) alternative business and information flow, 4) population growth and urbanization, and 5) markets and trade. The most important specific driver is exotic varieties and breeds among 90 specific drivers. The earthquake of 2015 was a major disaster having damaged much local agricultural diversity in more than 14 districts of Nepal. Expansion of area for agricultural production, the establishment of a new settlement and various types of developmental activities have caused a loss of many wild relatives of agricultural species and wild edible species. Threat to crop diversity is high in rice, banana, cucumber, and tomato due to replacement of old cultivars by modern varieties. Threat to rice bean, horse bean, and foxtail millet is due to low priority given by growers and consumers. Invasive alien species like Parthenium, Eupatorium spp. and Lantana camara have had a significant effect on diversity and ecosystem services. The least affected agrobiodiversity components are microbial genetic resources. The important drivers e.g., exotic genotypes, external inputs, monogenotype-based monoculture policy and climate changes, should be given due attention to control and manage for better management of agrobiodiversity.