Abstract - Rapid advancements in large language models have enabled significant progress in solving complex real-world problems using deep neural networks (DNN). However, the black box nature of these DNN models poses significant challenges when it comes to explaining their decisions to end users. In this study, we focus on the domain of natural language processing and investigate the application of explainable AI techniques to improve the understanding of decisions by deep neural networks, utilizing the Bidirectional Encoder Representations from Transformers (BERT). We employ two popular explainability tools, LIME and SHAP, to generate explanations for BERT's output to assess if their complementary use offers any supplementary explanation for AI decision making. The study employs statistical correlation analysis for LIME and SHAP explanations and determines that their complementary use will likely enhance end-user understanding of large language model decisions. By enhancing AI large language model explainability, this work contributes to bridging the gap between complex AI models and human comprehension of their decisions, fostering trust in AI applications.