Many signaling pathways regulate seed size through the development of endosperm and maternal tissues, which ultimately results in a range of variations in seed size or weight. Seed size can be determined through the development of zygotic tissues (endosperm and embryo) and maternal ovules. In addition, in some species such as rice, seed size is largely determined by husk growth. Transcription regulator factors are responsible for enhancing cell growth in the maternal ovule, resulting in seed growth. Phytohormones induce significant effects on entire features of growth and development of plants and also regulate seed size. Moreover, the vegetative parts are the major source of nutrients, including the majority of carbon and nitrogen-containing molecules for the reproductive part to control seed size. There is a need to increase the size of seeds without affecting the number of seeds in plants through conventional breeding programs to improve grain yield. In the past decades, many important genetic factors affecting seed size and yield have been identified and studied. These important factors constitute dynamic regulatory networks governing the seed size in response to environmental stimuli. In this review, we summarized recent advances regarding the molecular factors regulating seed size in Arabidopsis and other crops, followed by discussions on strategies to comprehend crops’ genetic and molecular aspects in balancing seed size and yield.