Effective learning outcome modeling is crucial to the success of learning evaluation in education. In the digital age, the movement towards online learning and computerized assessments has resulted in an explosion of structured and unstructured educational data (e.g., learners' problem-solving process data), which offers new opportunities for large-scale learning outcome modeling. Traditional psychometric models are of limited scalability and cannot adequately model item and learner features with incomplete and unstructured learner performance data. Existing advances in machine learning typically don't account for learners' problem-solving processes for learning outcome modeling. Leveraging the collaborative filtering approach used in recommender systems, we develop a general framework of deep learning-based collaborative filtering with process data for enhanced learning outcome modeling, which is named LogCF. LogCF is capable of modeling learner- and item-skill associations as well as predicting learners' item responses. In our experiments on two datasets of distinctive features, we demonstrate the superior predictive capacity of LogCF compared with other educational data mining and psychometric measurement models under different conditions of training/test partition ratios. In addition, we derive three variants of LogCF to examine whether item-skill associations learned or refined by LogCF are superior to the expert-specified ones. In addition, we also demonstrate the interpretability of learner- and item-skill associations learned by LogCF.