Abstract—In this paper, we present a comprehensive frameworkfor assessing the explainability of various XAI methods, such asLIME and SHAP, across multiple datasets and machine learningmodels, with the ultimate goal of creating a unified multidimensionalexplainability score. Our methodology focuses on threekey aspects of explainability: fidelity, simplicity, and stability.We leverage benchmarking experiments to systematically evaluatethese aspects and use the insights gained to construct an offlineknowledge base. This knowledge base captures the explainabilityscores for each registered model and serves as a valuable resourcefor context-dependent evaluation of explainability. By analyzingthe complementary characteristics and metadata of AI models,datasets, and XAI methods, the knowledge base will enable theestimation of explainability scores for previously unseen datasetsand models. Properties like fidelity, simplicity, and stability mayvary significantly based on the dataset, underlying model, anddomain expertise of the end user. We demonstrate our frameworkby applying it to three open-source datasets, discussing the implicationsof the obtained results in relation to the characteristics ofthe datasets. Our work contributes to the growing field of XAI byproviding a robust and versatile tool for evaluating and comparingthe explainability of various XAI methods, ultimately supportingthe development of more transparent and trustworthy AI systems.Index Terms—XAI, explainability score,