In this work we present an algebraic theory of multiple orthogonal polynomials. Our departure point is the three term recurrence relation, with matrix coefficients, satisfied by a sequence of vector multiple orthogonal polynomials. We give some characterizations of multiple orthogonal polynomials including recurrence relations, a Favard type theorem and a Christoffel-Darboux type formulas. An reinterpretation of the problems of Hermite-Pad´e approximation is presented.