With the development of edge devices and cloud computing, the question of how to accomplish machine learning and optimization tasks in a privacy-preserving and secure way has attracted increased attention over the past decade. As a privacy-preserving distributed machine learning method, federated learning (FL) has become popular in the last few years. However, the data privacy issue also occurs when solving optimization problems, which has received little attention so far. This survey paper is concerned with privacy-preserving optimization, with a focus on privacy-preserving data-driven evolutionary optimization. It aims to provide a roadmap from secure privacy-preserving learning to secure privacy-preserving optimization by summarizing security mechanisms and privacy-preserving approaches that can be employed in machine learning and optimization. We provide a formal definition of security and privacy in learning, followed by a comprehensive review of FL schemes and cryptographic privacy-preserving techniques. Then, we present ideas on the emerging area of privacy-preserving optimization, ranging from privacy-preserving distributed optimization to privacy-preserving evolutionary optimization and privacy-preserving Bayesian optimization (BO). We further provide a thorough security analysis of BO and evolutionary optimization methods from the perspective of inferring attacks and active attacks. On the basis of the above, an in-depth discussion is given to analyze what FL and distributed optimization strategies can be used for the design of federated optimization and what additional requirements are needed for achieving these strategies. Finally, we conclude the survey by outlining open questions and remaining challenges in federated data-driven optimization. We hope this survey can provide insights into the relationship between FL and federated optimization and will promote research interest in secure federated optimization.