This systematic review explores modern optimization methods for machine learning, distinguishing between gradient-based techniques using derivative information and population-based approaches employing stochastic search. Key innovations focus on enhanced regularization, adaptive control mechanisms, and biologically inspired strategies to address challenges like scaling to large models, navigating complex non-convex landscapes, and adapting to dynamic constraints. These methods underpin core ML tasks including model training, hyperparameter tuning, and feature selection. While significant progress is evident, limitations in scalability and theoretical guarantees persist, directing future work toward more robust and adaptive frameworks to advance AI applications in areas like autonomous systems and scientific discovery.