The validity and reliability of adaptive comparative judgements in the assessment of graphical capability.

Abstract

The valid and reliable assessment of capability is of paramount importance in education. Operationalizing assessment practices of divergent problems can be particularly challenging due to the variety of potential responses. This paper investigates the use of Adaptive Comparative Judgements (ACJ) in the assessment of graphical capability. A cohort of undergraduate Initial Technology Teacher Education (ITTE) students (N=128) participated in this study which involved completing a design task and subsequently assessing the work of their peers through ACJ and criterion referenced assessment. The performance from both methods was analysed and identified a high level of reliability for ACJ. Correlations between the criteria scores and ACJ parameter values suggest its validity as an assessment mechanism, however they also present the potential for additional variable to be influencing holistic judgements.