Unpacking decision making in comparative judgement: a stimulated think-aloud methodology to gain insight into young peoples' decision making

Abstract

While reliability has been the subject of much Comparative Judgement (CJ) research, understanding its validity, which and relates directly to the included judges and their decision making, is paramount. Understanding this decision-making better would add significantly to the formative use of CJ and its use in educational task-design. This paper reports on a pilot study exploring a novel methodology aiming to unpack judges’ decision-making. One 11-year-old student completed a CJ session on a selection of portfolios developed in response to an authentic design-task in STEM education. During this, a novel “stimulated think aloud protocol” was implemented, which was developed by synthesising aspects of traditional think-aloud-protocols with stimulated recall interviews. The approach is considered to have worked well as it was immediately evident that prompts were required to keep the participant on task and to continue verbalising their thoughts. As in this case the participant was younger, giving support in what to verbalise appeared necessary. The approach was possibly more useful due to the age of the participant. Limitations exist in that the stimulating prompts could influence participant decision-making if they provoke reflection which otherwise would not have occurred in an undisrupted CJ judging session.