-
Research Matters 37 - Foreword
Oates, T. (2024). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 37, 4.
The explosion of activity around large language models since the release of ChatGPT in November 2022 has pushed some very important assessment issues aside. It is vital not to forget them. The principles and criteria for high quality assessment - whether low or high stakes in character - have been carefully accumulated through experience and research. It's extremely important to remember that innovation in assessment should rest on solid foundations.
Download
-
Research Matters 37 - Editorial
Bramley, T. (2024). Editorial.Research Matters: A Cambridge University Press & Assessment publication, 37, 5.
Our first article describes the technologies collectively known as "extended reality" and considers opportunities and challenges for using them in teaching and assessing mathematics. The second reports on a study where three undergraduates were asked to use ChatGPT to assist with writing essays and then interviewed about their approach. Our third article considers the difficult issues that arise when comparing curriculum documents with the aim of making claims about comparability of different curricula. Our fourth article explores the extent to which data (specifically whether a response was missing or not) can support inferences about whether students were under time pressure in paper-based GCSE examination components, and whether exams in some subjects were more "speeded" than others. Our final article presents a historical overview of the Centre for Evaluation and Monitoring (CEM), acquired by Cambridge in 2019 but now celebrating more than 40 years since its creation.
Download
-
Extended Reality (XR) in mathematics assessment: A pedagogical vision
Li, X. (2024). Extended Reality (XR) in mathematics assessment: A pedagogical vision. Research Matters: A Cambridge University Press & Assessment publication, 37, 6-23. https://doi.org/10.17863/CAM.106031
Extended reality (XR) – encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR) – emerges as a potential transformative tool in educational realms. This article explores the potential of XR in facilitating mathematics assessments; it proposes a list of mathematical topics that could be effectively mediated by XR’s immersive and interactive features. Additionally, it discusses some major challenges which could be barriers to the widespread adoption of XR in educational contexts and sets out a research agenda for further investigation.
Download
-
Does ChatGPT make the grade?
Brady, J., Kuvalja, M., Rodrigues, A., & Hughes, S. (2024). Does ChatGPT make the grade? Research Matters: A Cambridge University Press & Assessment publication, 37, 24-39. https://doi.org/10.17863/CAM.106034
This study explores undergraduate students’ use of ChatGPT when writing essays. Three students were tasked with writing two essays each for a coursework component for a Cambridge qualification facilitated by access to ChatGPT. After writing the essays, they participated in semi-structured interviews about their experiences of using the technology. Researchers compared the transcript of the chatlog between the students and ChatGPT with the submitted essays. Analysis showed that the students relied on ChatGPT outputs to different extents, although they followed a similar process of engagement. The students shared their misgivings and points of appreciation for the technology.
Download
-
How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictions
Rushton, N., Majewska, D., & Shaw, S. (2024). How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictions. Research Matters: A Cambridge University Press & Assessment publication, 37, 40-56. https://doi.org/10.17863/CAM.106032
Curriculum mapping is a comparability method that facilitates comparisons of content within multiple settings (usually multiple jurisdictions or specifications) and enables claims to be made about those curriculums/jurisdictions. Although curriculum maps have been published, there is little academic literature about the process of constructing and using them. Our study extends the literature by considering the different types of comparisons that can be made from curriculum maps: content coverage, placement, depth, and breadth. We also consider how these comparisons are affected by structural differences in the curricula or using a sub-set of the content.
We use our mapping of mathematics in the US Common Core State Standards (CCSS) and the national curriculum in England to explore this. The CCSS for mathematical practice are common to all grades; we mapped these standards against the content for individual years in the national curriculum. The CCSS for mathematical content are set out by grade; we mapped a subset of this content to the national curriculum.
Our mapping shows that it is possible to map curricula and make meaningful comparisons despite structural differences and content limitations. However, this affected the types of comparisons that we could carry out and the claims that we could make.
Download
-
Exploring speededness in pre-reform GCSEs (2009 to 2016)
Walland, E. (2024). Exploring speededness in pre-reform GCSEs (2009 to 2016). Research Matters: A Cambridge University Press & Assessment publication, 37, 57-73. https://doi.org/10.17863/CAM.106035
GCSE examinations (taken by students aged 16 years in England) are not intended to be speeded (i.e. to be partly a test of how quickly students can answer questions). However, there has been little research exploring this. The aim of this research was to explore the speededness of past GCSE written examinations, using only the data from scored responses to items from a sample of 340 GCSE components. Speededness was calculated as the average (mean) percentage marks lost from the longest string of unanswered items at the end of each student’s examination paper. The potential impact of student ability on examination completion patterns was taken into account. The data suggested that most GCSEs analysed were unlikely to have been speeded. This method of exploring the speededness of exams using only scored responses has potential (although there are limitations), and it can flag potentially problematic components for further investigation.
Download
-
A Short History of the Centre for Evaluation and Monitoring (CEM)
Jellis, C. (2024). A short history of the Centre for Evaluation and Monitoring (CEM). Research Matters: A Cambridge University Press & Assessment publication, 37, 74-83. https://doi.org/10.17863/CAM.106033
The Centre for Evaluation and Monitoring (CEM), based in the North of England, recently celebrated its 40th birthday. Arising from an evaluation project at Newcastle University, and a subsequent move to Durham University, it rapidly grew in scope and influence, developing a series of highly regarded school assessments. For a relatively small organisation, its influence was seen across the world, resulting in outreach centres in New Zealand, Australia, and Hong Kong. Since being acquired by Cambridge University Press & Assessment in 2019, it has established itself in a unique role within the wider Cambridge organisation due mainly to its development of computer adaptive assessments for use in schools. This article documents the rise of CEM, from its early successes to its adoption of new ideas in educational assessment and supporting technology until the present day. However, CEM’s development was not without its controversies, and these too make fascinating reading when set against the background of 40 years of ever changing educational policies.
Download
-
Research Matters issue 37 - Research News
Bowett, L. (2024). Research News. Research Matters: A Cambridge University Press & Assessment publication, 37, 84-86.
A summary of recent conferences, reports, blogs and research articles published since the last issue of Research Matters.
Download