Within a framework of noncommutative geometry, we develop an analog of (pseudo-) Riemannian geometry on finite and discrete sets. On a finite set, there is a counterpart of the continuum metric tensor with a simple geometric interpretation. The latter is based on a correspondence between first order differential calculi and digraphs (the vertices of the latter are given by the elements of the finite set). Arrows originating from a vertex span its (co)tangent space. If the metric is to measure length and angles at some point, it has to be taken as an element of the left-linear tensor product of the space of 1-forms with itself, and not as an element of the (nonlocal) tensor product over the algebra of functions, as considered previously by several authors. It turns out that linear connections can always be extended to this left tensor product, so that metric compatibility can be defined in the same way as in continuum Riemannian geometry. In particular, in the case of the universal differential calculus on a finite set, the Euclidean geometry of polyhedra is recovered from conditions of metric compatibility and vanishing torsion. In our rather general framework (which also comprises structures which are far away from continuum differential geometry), there is, in general, nothing like a Ricci tensor or a curvature scalar. Because of the nonlocality of tensor products (over the algebra of functions) of forms, corresponding components (with respect to some module basis) turn out to be rather nonlocal objects. But one can make use of the parallel transport associated with a connection to “localize” such objects, and in certain cases there is a distinguished way to achieve this. In particular, this leads to covariant components of the curvature tensor which allow a contraction to a Ricci tensor. Several examples are worked out to illustrate the procedure. Furthermore, in the case of a differential calculus associated with a hypercubic lattice we propose a new discrete analogue of the (vacuum) Einstein equations.