Living creatures improve their adaptation capabilities to a changing world by means of two orthogonal processes: evolution and lifetime learning. Within Artificial Intelligence, both mechanisms inspired the development of non-orthodox problem solving tools, namely Genetic and Evolutionary Algorithms (GEAs) and Artificial Neural Networks (ANNs). Several local search gradient-based methods have been developed for ANN training, with considerable success; however, in some situations, such procedures may lead to local minima. Under this scenario, the combination of evolution and learning techniques, may lead to better results (e.g., global optima). Comparative tests on several Machine Learning tasks attest this claim.