Standing on the shoulders of giants: Online formative assessments as the foundation for predictive learning analytics models

As universities around the world have begun to use learning management systems (LMSs), more learning data have become available to gain deeper insights into students' learning processes and make data-driven decisions to improve student learning. With the availability of rich data extracted from the LMS, researchers have turned much of their attention to learning analytics (LA) applications using educational data mining techniques. Numerous LA models have been proposed to predict student achievement in university courses. To design predictive LA models, researchers often follow a data-driven approach that prioritizes prediction accuracy while sacrificing theoretical links to learning theory and its pedagogical implications. In this study, we argue that instead of complex variables (e.g., event logs, clickstream data, timestamps of learning activities), data extracted from online formative assessments should be the starting point for building predictive LA models. Using the LMS data from multiple offerings of an asynchronous undergraduate course, we analysed the utility of online formative assessments in predicting students' final course performance. Our findings showed that the features extracted from online formative assessments (e.g., completion, timestamps and scores) served as strong and significant predictors of students' final course performance. Scores from online formative assessments were consistently the strongest predictor of student performance across the three sections of the course. The number of clicks in the LMS and the time difference between first access and due dates of formative assessments were also significant predictors. Overall, our findings emphasize the need for online formative assessments to build predictive LA models informed by theory and learning design. Practitioner notes What is already known about this topic Higher education institutions often use learning analytics for the early identification of low-performing students or students at risk of dropping out. Most predictive models in learning analytics rely on immutable student characteristics (e.g., gender, race and socioeconomic status) and complex variables extracted from log data within a learning management system. Prioritizing prediction accuracy without theory orientation often yields "black-box" models that fail to inform educators on what remedies need to be taken to improve student learning. What this paper adds Predictive models in learning analytics should consider learning theory, pedagogy and learning design to identify key predictors of student learning. Online formative assessments can be a starting point for building predictive models that are not only accurate but also provide educators with actionable insights on how student learning can be improved. Time-related and score-related features extracted from online formative assessments are particularly useful for predicting students' course performance. Implications for practice and/or policy This study provides strong evidence for using online formative assessments as the foundation for predictive models in learning analytics. Student data from online formative assessments can help educators provide students with feedback while informing future formative assessment cycles. Higher education institutions should avoid the hype around complex data from learning management systems and instead rely on effective learning tools such as online formative assessments to revolutionize the use of learning analytics.