In this paper, we present Montimage AI Platform (MAIP), a new GUI-based deep learning framework for malicious traffic detection and classification combined with its ability of explaining the decision of the model. We employ popular XAI methods to interpret the prediction of the developed deep learning model. Furthermore, we perform adversarial attacks to assess the accountability and robustness of our model via different quantifiable metrics. We perform extensive experiments with both public and private network traffic. The experimental results demonstrate that our model achieves high performance and robustness, and its outcomes align closely with the domain knowledge.