Differential Performance of Computerized Adaptive Testing in Students With and Without Disabilities

Ebenbeck, Nikola (7. February 2024): Differential Performance of Computerized Adaptive Testing in Students With and Without Disabilities. A Simulation Study. In: Journal of Special Education Technology [Forthcoming]

Abstract

Technologies that enable individualization for students have significant potential in special education. Computerized Adaptive Testing (CAT) refers to digital assessments that automatically adjust their difficulty level based on students' abilities, allowing for personalized, efficient, and accurate measurement. This article examines whether CAT performs differently for students with and without special educational needs (SEN). Two simulation studies were conducted using a sample of 709 third-grade students from general and special schools in Germany, who took a reading test. The results indicate that students with SEN were assessed with fewer items, reduced bias, and higher accuracy compared to students without SEN. However, measurement accuracy decreased, and test length increased for students whose abilities deviated more than two SD from the norm. We discuss potential adaptations of CAT for students with SEN in the classroom, as well as the integration of CAT with AI-supported feedback and tailored exercises within a digital learning environment.

• Item Type: Journal article
• Keywords: computerized adaptive testing; special educational needs; individualization; assessment
• Subjects: 300 Social sciences > 370 Education
• Annotation: First published online February 7, 2024
• Language: English

References

Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (2009). Item selection rules in computerized adaptive testing: Accuracy and security. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(1), 7.

Birnbaum, A. (1969). Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, 6(2), 258-276.

Chen, X., Zou, D., Xie, H., Cheng, G., & Liu, C. (2022). Two decades of artificial intelligence in education. Educational Technology & Society, 25(1), 28-47.

Choi, Y., & McClenen, C. (2020). Development of adaptive formative assessment system using computerized adaptive testing and dynamic Bayesian networks. Applied Sciences, 10(22), 8196.

Davey, T., Pitoniak, J., & Slater, S. C. (2015). Designing Computerized Adaptive Tests. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.), Handbook of Test Development (2nd ed., pp. 467–484). Routledge.

Ebenbeck, N. (2023). Computerized Adaptive Testing in Inclusive Education. Dissertation. Publikationsserver der Universität Regensburg.

Ebenbeck, N., Rieser, J., Jungjohann, J., & Gebhardt, M. (2022). How the existence of special schools affects the placement of students with special needs in inclusive primary schools. Journal of Research in Special Educational Needs, 22(3), 274-287.

Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press.

Forkmann, T., Kroehne, U., Wirtz, M., Norra, C., Baumeister, H., Gauggel, S., ... & Boecker, M. (2013). Adaptive screening for depression—Recalibration of an item bank for the assessment of depression in persons with mental and somatic diseases and evaluation in a simulated computer-adaptive test environment. Journal of Psychosomatic Research, 75(5), 437-443.

Georgiadou, E., Triantafillou, E., & Economides, A. A. (2006). Evaluation parameters for computer-adaptive testing. British Journal of Educational Technology, 37(2), 261–278. https://doi.org/10.1111/j.1467-8535.2005.00525.x

Gibbons, R. D., Weiss, D. J., Kupfer, D. J., Frank, E., Fagiolini, A., Grochocinski, V. J., ... & Immekus, J. C. (2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59(4), 361-368.

Green, B. F. (1983). The promise of tailored tests. In Principles of Modern Psychological Measurement (pp. 69-80).

He, W., & Reckase, M. D. (2014). Item pool design for an operational variable-length computerized adaptive test. Educational and Psychological Measurement, 74(3), 473-494.

Hoyer, R. S., Elshafei, H., Hemmerlin, J., Bouet, R., & Bidet‐Caulet, A. (2021). Why are children so distractible? Development of attention and motor control from childhood to adulthood. Child development, 92(4), e716-e737.

Huang, H.-T. D., Hung, S.-T. A., Chao, H.-Y., Chen, J.-H., Lin, T.-P., Shih, & C.-L. (2022). Developing and validating a computerized adaptive testing system for measuring the English proficiency of Taiwanese EFL university students. Language Assessment Quarterly, 19(2), 162–188. https://doi.org/10.1080/15434303.2021.1984490

Jungjohann, J., Ebenbeck, N., Liebers, K., Diehl, K., & Gebhardt, M. (submitted). Requirements and development of a reading screening (LES-IN) for third graders in inclusive classrooms.

Kalender, I., & Berberoglu, G. (2017). Can computerized adaptive testing work in students’ admission to higher education programs in Turkey? Educational Sciences: Theory & Practice, 17(2).

Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359-375. DOI: 10.1207/s15324818ame0204_6

Lancaster, S., Mellard, D., & Hoffman, L. (2001). Experiences of students with disabilities in selected community and technical colleges. In The individual accommodation model: Accommodating students with disabilities in postsecondary settings. University of Kansas.

Liebers, K., Kanold, E., & Junger, R. (2019). Digitale Lernstandsanalysen in der inklusiven Grundschule? [Digital learning assessments in inclusive primary schools?] In S. Bartusch, C. Klektau, T. Simon, & Teumer, S. Weidermann, A. (Eds.), Lernprozesse Begleiten [Accompanying learning processes] (pp. 209–221). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-21924-6_16

Ludewig, U., Trendtel, M., Schlitter, T., & McElvany, N. (2022). Adaptives Testen von Textverständnis in der Grundschule [Adaptive testing of text comprehension in elementary school]. Diagnostica. Magis, D., Raiche, G., & Magis, M. D. (2018). Package ‘catR’. R package, version 3. Retrieved from http://cran.stat.unipd.it/web/packages/catR/catR.pdf

Magis, D., Yan, D., & Von Davier, A. A. (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer. Marino, M. T., Vasquez, E., Dieker, L., Basham, J., & Blackorby, J. (2023). The future of artificial intelligence in special education technology. Journal of Special Education Technology, 0(0). https://doi.org/10.1177/01626434231165977

Martin, A. J., & Lazendic, G. (2018). Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of educational psychology, 110(1), 27.

Meijer, R. R., & Nering, M. L. (1999). Computerized Adaptive Testing: Overview and introduction. Applied Psychological Measurement, 23(3), 187–194. https://doi.org/10.1177/01466219922031310

Michiel H. A., Vorst, H. C. M., & Mellenbergh, G. J. (2008). Computerized Adaptive Testing of personality traits. Journal of Psychology, 216(1), 12–21.

Mills, C. N., & Stocking, M. L. (1995). Practical issues in large‐scale high‐stakes Computerized Adaptive Testing. ETS Research Report Series, 1995(2), i-28.

Mizumoto, A., Sasao, Y., & Webb, S. A. (2019). Developing and Evaluating a Computerized Adaptive Testing Version of the Word Part Levels Test. Language Testing, 36(1), 101–123. https://doi.org/10.1177/0265532217725776

Oladele, J. I., & Ndlovu, M. (2021). A review of standardized assessment development procedure and algorithms for computer adaptive testing: Applications and relevance for fourth industrial revolution. International Journal of Learning, Teaching and Educational Research, 20(5), 1-17.

Parshall, C. G., Davey, T., & Pashley, P. J. (2000). Innovative Item Types for Computerized Testing. In W. J. van der Linden & G. A. W. Glas (Eds.), Computerized Adaptive Testing: Theory and Practice (pp. 129–148). Springer Netherlands. https://doi.org/10.1007/0-306-47531-6_7

R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/ Reckase, M. D. (2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127.

Reed, D. K., Martin, E., Hazeltine, E., & McMurray, B. (2020). Students’ perceptions of a gamified reading assessment. Journal of Special Education Technology, 35(4), 191–203. https://doi.org/10.1177/0162643419856272

Rudner, L. M. (1998). An on-line, interactive, computer adaptive testing tutorial. ERIC Clearinghouse on Assessment and Evaluation. Şahin, A., & Anıl, D. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1).

Sands, W. A., Waters, B. K., & McBride, J. R. (Eds.). (1997). Computerized adaptive testing: From inquiry to operation. American Psychological Association. https://doi.org/10.1037/10244-000

Schaper, N. (2009). Online-Tests aus diagnostisch-methodischer Sicht [Online tests from a diagnostic-methodological perspective]. In H. Steiner (Ed.), Online-Assessment (pp. 17–36). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-78919-2_2.

Schildkamp, K., Lai, M. K., & Earl, L. (Eds.). (2012). Data-based decision making in education: Challenges and opportunities. Segall, D. O. (2005). Computerized adaptive testing. Encyclopedia of social measurement, 1, 429-438.

Stevens, S. A. (2001). Test anxiety and beliefs about testing in college students with and without learning disabilities. [Doctoral dissertation, University of Massachusetts Amherst]. https://doi.org/10.7275/61f5-y146

Stone, E., & Davey, T. (2011). Computer-adaptive testing for students with disabilities: A review of the literature. ETS Research Report Series, 2011(2), i-24.

Swanson, S., & Howell, C. C. (1996). Test anxiety in adolescents with learning disabilities and behavior disorder. Exceptional Children, 62, 389–397. https://doi.org/10.1177/001440299606200501

Thompson, N. A., & Weiss, D. A. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research, and Evaluation, 16(1), 1.

Thurlow, M. L., Thompson, S. J., & Lazarus, S. S. (2011). Considerations for the administration of tests to special needs students: Accommodations, modifications, and more. In T. M. Haladyna & S. M. Downing (Eds.), Handbook of test development (pp. -). Routledge.

Tian, J., Miao, D., Zhu, X., & Gong, J. (2007). An introduction to Computerized Adaptive Testing. US-China Education Review, 4(1), 72–81.

Turner, M. I., Van Norman, E. R., & Hojnoski, R. L. (2022). An independent evaluation of the diagnostic accuracy of a computer adaptive test to predict proficiency on an end-of-year high-stakes assessment. Journal of Psychoeducational Assessment, 40(7), 911-916.

Van der Linden, W. J., & Glas, C. A. (Eds.). (2000). Computerized Adaptive Testing: Theory and practice. Springer Science & Business Media.

Vidal Rodeiro, C., & Macinska, S. (2022). Equal opportunity or unfair advantage? The impact of test accommodations on performance in high-stakes assessments. Assessment in Education: Principles, Policy & Practice, 29(4), 462-481. https://doi.org/10.1080/0969594X.2022.2121680

Walter, T., & Schuhfried, G. (2004). Computergestützte psychologische Diagnostik [Computer-assisted psychological assessment]. In G. Mehta (Ed.), Springer eBook Collection Humanities, Social Science. Die Praxis der Psychologie: Ein Karriereplaner [The Practice of Psychology: A Career Planner] (pp. 265–272). Springer Vienna. https://doi.org/10.1007/978-3-7091-0571-9_21

Weiss, D. J. (2004). Computerized Adaptive Testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37(2), 70-84.