Item Response Theory-Based Gaming Detection

Item Response Theory-Based Gaming Detection

Item Response Theory-Based Gaming Detection

By Yun Huang, Steven Dang, J. Elizabeth Richey, Michael Asher, Nikki G. Lobczowski, Danielle Chine, Elizabeth A. McLaughlin, Judith M. Harackiewicz, Vincent Aleven and Kenneth Koedinger

Abstract

Gaming the system, a behavior in which learners exploit a system’sproperties to make progress while avoiding learning, has frequentlybeen shown to be associated with lower learning. However, whenwe applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detectedgaming was not associated with learning, challenging its constructvalidity. Our iterative exploratory data analysis suggested thatsome contextual factors that varied across and within conditionsmight contribute to this lack of association. We present a latentvariable model, item response theory-based gaming detection(IRT-GD), that accounts for contextual factors and estimates latentgaming tendencies as the degree of deviation from normativebehaviors across contexts. Item response theory models, widely used in knowledge assessment, account for item difficulty inestimating latent student abilities: students are estimated as havinghigher ability when they can get harder items correct than whenthey only get easier items correct. Similarly, IRT-GD accounts forcontextual factors in estimating latent gaming tendencies: studentsare estimated as having a higher gaming tendency when they game in less commonly gamed contexts than when they only game inmorecommonly gamed contexts. IRT-GD outperformed theoriginal detector on three datasets in terms of the association withlearning. IRT-GD also more accurately revealed interventioneffects on gaming and revealed a correlation between gaming andperceived competence in math. Our approach is not only useful forothers wanting to apply a gaming assessment in their context but isalso generally applicable in creating robust behavioral measures.”

Reference

Huang, Y., Dang, S., Richey, J., Asher, M., Lobczowski, N., Chine, D., . . . Koedinger, K. (2022, July). Item response theory-based gaming detection. Retrieved September 14, 2022, from https://educationaldatamining.org/EDM2022/proceedings/2022.EDM-long-papers.22/index.html

Keyword

Gaming the system, item response theory, behavior modeling, research