This survey presents recent progress on using machine learning techniques to improve query optimizers in database systems. Centering around a generic paradigm of learned query optimizers, this survey covers several lines of effort on rebuilding or aiding important components in query optimizers (i.e., cardinality estimators, cost models, and plan enumerators) with machine learning. We introduce some important machine learning tools developed recently, which are useful for query optimization, and how they are adapted for sub-tasks of query optimization. This survey is for readers who are already familiar with query optimization and are eager to understand what machine learning techniques can be helpful and how to apply them with examples and necessary details, or for machine learning researchers who want to expand their research agendas to helping database systems with machine learning techniques. Some open research challenges are also discussed with the goal of making learned query optimizers truly applicable in production.