Emma Walland

Chambers, L., Walland, E., & Ireland, J. (2024). Comparing music recordings using Pairwise Comparative Judgement: Exploring the judge experience Research Matters: A Cambridge University Press & Assessment publication, 38, 82-98. https://doi.org/10.17863/CAM.111630

Comparative Judgement (CJ) is traditionally and primarily used to compare written texts. In this study we explored whether we could extend its use to comparing audio files. We used GCSE Music portfolios which contained a mix of audio recordings, musical scores and text documents. Fifteen judges completed two exercises: one comparing musical compositions and one comparing musical performances. For each exercise, each judge compared 80 pairs of portfolios. Once judges had finished both exercises, they completed a questionnaire about their views and experiences of the method. Here, we present the judges’ perceptions of using CJ in this context with reference to the Dimensions of judge decision-making model (Leech & Chambers 2022). We also compare the findings to those from text-based CJ studies.

Leech, T., & Chambers, L. (2022). How do judges in Comparative Judgement exercises make their judgements? Research Matters: A Cambridge University Press & Assessment publication, 33, 31–47.

• Foreword Tim Oates
• Editorial Victoria Crisp
• Troubleshooting in emergency education settings: What types of strategies did schools employ during the COVID-19 pandemic and what can they tell us about schools’ adaptability, values and crisis-readiness?Filio Constantinou
• How long should a high stakes test be?Tom Benton
• Core Maths: Who takes it, what do they take it with, and does it improve performance in other subjects?Tim Gill
• Does typing or handwriting exam responses make any difference? Evidence from the literatureSanti Lestari
• Comparing music recordings using Pairwise Comparative Judgement: Exploring the judge experienceLucy Chambers, Emma Walland and Jo Ireland
• Research NewsLisa Bowett

Walland, E. (2024). Exploring speededness in pre-reform GCSEs (2009 to 2016). Research Matters: A Cambridge University Press & Assessment publication, 37, 57-73. https://doi.org/10.17863/CAM.106035

GCSE examinations (taken by students aged 16 years in England) are not intended to be speeded (i.e. to be partly a test of how quickly students can answer questions). However, there has been little research exploring this. The aim of this research was to explore the speededness of past GCSE written examinations, using only the data from scored responses to items from a sample of 340 GCSE components. Speededness was calculated as the average (mean) percentage marks lost from the longest string of unanswered items at the end of each student’s examination paper. The potential impact of student ability on examination completion patterns was taken into account. The data suggested that most GCSEs analysed were unlikely to have been speeded. This method of exploring the speededness of exams using only scored responses has potential (although there are limitations), and it can flag potentially problematic components for further investigation.

• Foreword Tim Oates
• Editorial Tom Bramley
• Extended Reality (XR) in mathematics assessment: A pedagogical visionXinyue Li
• Does ChatGPT make the grade?Jude Brady, Martina Kuvalja, Alison Rodrigues, Sarah Hughes
• How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictionsNicky Rushton, Dominika Majewska, Stuart Shaw
• Exploring speededness in pre-reform GCSEs (2009 to 2016)Emma Walland
• A Short History of the Centre for Evaluation and Monitoring (CEM)Chris Jellis
• Research NewsLisa Bowett

Walland, E., & Leech, T. (2023, November 1–4). How are GCSE grades used in post-16 admissions decisions in England? [Paper presentation]. Annual conference of the Association for Educational Assessment – Europe (AEA-Europe), Malta. https://2023.aea-europe.net/

• Foreword Tim Oates
• Editorial Tom Bramley
• A summary of OCR’s pilots of the use of Comparative Judgement in setting grade boundaries Tom Benton, Tim Gill, Sarah Hughes, Tony Leech
• How do judges in Comparative Judgement exercises make their judgements? Tony Leech, Lucy Chambers
• Judges' views on pairwise Comparative Judgement and Rank Ordering as alternatives to analytical essay marking Emma Walland
• The concurrent validity of Comparative Judgement outcomes compared with marks Tim Gill
• How are standard-maintaining activities based on Comparative Judgement affected by mismarking in the script evidence? Joanna Williamson
• Moderation of non-exam assessments: is Comparative Judgement a practical alternative? Carmen Vidal Rodeiro, Lucy Chambers
• Research News Lisa Bowett

Walland, E. (2022). Judges’ views on pairwise Comparative Judgement and Rank Ordering as alternatives to analytical essay marking. Research Matters: A Cambridge University Press & Assessment publication, 33, 48–67.

In this article, I report on examiners’ views and experiences of using Pairwise Comparative Judgement (PCJ) and Rank Ordering (RO) as alternatives to traditional analytical marking for GCSE English Language essays. Fifteen GCSE English Language examiners took part in the study. After each had judged 100 pairs of essays using PCJ and eight packs of ten essays using RO, I collected data on their experiences and views of the methods through interviews and questionnaires. I analysed the data using thematic content analysis.

The findings highlight that, if the methods were to be used as alternatives to marking, examiners and other stakeholders would need reassurance that the methods are fair, valid and reliable. Examiners would also need more training and support to help them to judge holistically. The lack of detail about how judgements are made using these methods is a concern worth following up and addressing before implementation.

Walland, E. and Shaw, S. (2020). Using e-portfolios to capture and assess transversal skills: tensions in theory and praxis. Presented at the International Conference of Education, Research and Innovation (ICERI), Online, 9-11 November 2020.

The video can be viewed here.

Walland, E. and Darlington, E. (2019). Teacher decision-making on post-16 provision in response to reform. Presented at the annual conference of the British Educational Research Association, University of Manchester, Manchester, UK, 10-12 September 2019. The slides and commentary can be viewed here.

Elliott, G., Suto, I. and Walland, E. (2019). What makes researchers anxious? It’s Time to Talk about talking about research. Research Matters: A Cambridge Assessment publication, 27, 26-32.

Increasingly, workplaces provide accessible places and events for employees to discuss mental health issues. In the course of discussions in our organisation, researchers raised the issue of workplace activities that cause them anxiety – top of the list being giving presentations. This article describes a workshop held with researchers at Cambridge Assessment to encourage the discussion to develop more widely. We reflect upon themes about workplace anxiety identified by the workshop participants, relate the themes to the literature, and consider practical techniques for reducing the impact of anxiety. Our aim is that this study will be helpful in enabling other researchers to become more open in discussing the aspects of their role which cause anxiety. In turn, this will facilitate finding ameliorative solutions to the issues raised, to the mutual benefit of individuals and organisations.