Many applications require the identification of persons in video. However, the set of persons of interest is not always known in advance, e.g., in applications for media production and archiving. Additional training samples may be added during the analysis, or groups of faces of one person may need to be identified retrospectively. In order to avoid re-running the face recognition, we propose an approach that supports fast incremental training based on a state of the art face detection and recognition pipeline using CNNs and an online random forest as a classifier. We also describe an algorithm to use the incremental training approach to automatically train classifiers for unknown persons, including safeguards to avoid noise in the training data. We show that the approach reaches state of the art performance on two datasets when using all training samples, but performs better with few or even only one training sample.