Integrating multi-modal patient data to support personalized medicine has gained a lot of interest across different health domains over the past decade. Addressing this challenge requires the development and implementation of an informed, evidence-based AI-driven decision-support system continuously maintained and updated to align with the latest clinical guidelines. A key challenge to ensure its real-life adoption lies in translating the outcomes of complex AI-driven data integration and modeling into a form easily understood by the clinical audience. To ensure explainability, knowledge graphs have emerged as data models integrating multi-omics data sources and representing them as interconnected networks. Knowledge graphs offer a framework which AI models can progressively refine, highlighting the most influential features and relationships facilitating transparency of complex interactions and interdependencies. In this perspective we present major components and challenges upon developing a knowledge-based explainable AI system. Additionally, we showcase a current effort undertaken by the Knowledge at the Tips of your Fingers (KATY) consortium to develop the infrastructure for an explainable system supporting best treatment decision for a renal cancer patient.