IDENTIFYING STUDENT PROFILES WITHIN ONLINE JUDGE SYSTEMS USING EXPLAINABLE ARTIFICIAL INTELLIGENCE

Abstract

 Online Judge (OJ) systems are typically considered within programming-related courses as they yield fast and 
objective assessments of the code developed by the students. Such an evaluation generally provides a single decision 
based on a rubric, most commonly whether the submission successfully accomplished the assignment. Nevertheless, 
since in an educational context such information may be deemed insufficient, it would be beneficial for both the 
student and the instructor to  receive additional feedback about the overall development of the task. This work aims 
to tackle this limitation by considering the further exploitation of the information gathered by the OJ and automatically 
inferring feedback for both the student and the instructor. More precisely, we consider the use of learning-based 
schemes—particularly, Multi-Instance Learning and classical Machine Learning formulations—to model student 
behaviour. Besides, Explainable Artificial Intelligence is contemplated to provide human-understandable feedback. 
The proposal has been evaluated considering a case of study comprising 2,500 submissions from roughly 90 different 
students from a programming-related course in a Computer Science degree. The results obtained validate the proposal: 
the model is capable of significantly predicting the user outcome (either passing or failing the assignment) solely 
based on the behavioural pattern inferred by the submissions provided to the OJ. Moreover, the proposal is able to 
identify prone-to-fail student groups and profiles as well as other relevant information, which eventually serves as 
feedback to both the student and the instructor.