In medical image analysis a ground truth to compare results against is of vital importance. This ground truth is often obtained from human experts. The aim of this paper is to discuss the problem related to the use of markings made by an expert panel. As a partial solution, we propose a method to relate markings to each other in order to establish levels of agreement. By using this method we can assess the performance of, for instance, segmentation algorithms.