This chapter is aimed at establishing the conceptual foundation of the relevant aspects of machine learning and statistics on which the book rests. This very brief overview is in no way exhaustive. Rather, our main aim is to elucidate the relationship of these concepts to the performance evaluation of learning algorithms. The chapter is composed of two parts. The first part discusses concepts most specific to machine learning; the second part focuses on the statistical elements. Even though these may seem like two disparate parts, they are not entirely independent. We try to highlight the relationship between the concepts discussed in one field to the problems at hand in the other. Let us start with a brief discussion of the important concepts of machine learning. Machine Learning Overview Learning is the human process that allows us to acquire the skills necessary to adapt to the multitude of situations that we encounter throughout our lives. As human beings, we rely on many different kinds of learning processes at different stages to acquire different functionalities. We learn a variety of different skills, e.g., motor, verbal, mathematical, and so on. Moreover, the learning process differs with the situations and time, e.g., learning how to speak as a toddler is different from learning similar skill sets in a given profession. Variations in learning are also visible in terms of contexts and the related tools, e.g., classroom learning differs from social contexts, rote learning may be more suitable for memorizing but differs from learning how to reason.