This paper describes our experience in combining a large corpus of material that was classified manually over decades of Art His- torical and Book History research using the ICONCLASS4 subject clas- sification system with heuristics based on a current state-of-the-art neu- ral network (CLIP from OpenAI), leveraging visual similarity to provide suggestions for automated classification of cultural heritage content. The effectiveness of the approach is demonstrated through an evaluation of the underlying extreme multi-label classification problem.