Because of the multi-symptomatic nature of sepsis, its prediction is challenging as it requires considering subtle changes in multiple monitored variables across time. Recent works based on deep neural networks improved the prediction performance, but still suffer from poor interpretability. Critical for healthcare applications, we propose to improve this aspect by separating time variables in the convolution layers of a sepsis prediction network. We reveal the improvement in interpretability capacity with the use of gradient-based attributions on high-level intermediate features and through a metric correlating the variable attribution with the prediction for perturbed pathologic samples. With 171,945 patients from the MIMIC-IV database, we demonstrate that our method not only maintains classification performances at similar network parameters count, but also substantially improves the faithfulness of per-variable attributions.