Nicky Rushton

Rushton, N., Majewska, D., & Shaw, S. (2024). How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictions. Research Matters: A Cambridge University Press & Assessment publication, 37, 40-56. https://doi.org/10.17863/CAM.106032

Curriculum mapping is a comparability method that facilitates comparisons of content within multiple settings (usually multiple jurisdictions or specifications) and enables claims to be made about those curriculums/jurisdictions. Although curriculum maps have been published, there is little academic literature about the process of constructing and using them. Our study extends the literature by considering the different types of comparisons that can be made from curriculum maps: content coverage, placement, depth, and breadth. We also consider how these comparisons are affected by structural differences in the curricula or using a sub-set of the content.

We use our mapping of mathematics in the US Common Core State Standards (CCSS) and the national curriculum in England to explore this. The CCSS for mathematical practice are common to all grades; we mapped these standards against the content for individual years in the national curriculum. The CCSS for mathematical content are set out by grade; we mapped a subset of this content to the national curriculum.

Our mapping shows that it is possible to map curricula and make meaningful comparisons despite structural differences and content limitations. However, this affected the types of comparisons that we could carry out and the claims that we could make.

• Foreword Tim Oates
• Editorial Tom Bramley
• Extended Reality (XR) in mathematics assessment: A pedagogical visionXinyue Li
• Does ChatGPT make the grade?Jude Brady, Martina Kuvalja, Alison Rodrigues, Sarah Hughes
• How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictionsNicky Rushton, Dominika Majewska, Stuart Shaw
• Exploring speededness in pre-reform GCSEs (2009 to 2016)Emma Walland
• A Short History of the Centre for Evaluation and Monitoring (CEM)Chris Jellis
• Research NewsLisa Bowett

Rushton, N., & Lestari, S. (2023, November 1–4). COVID-19 related changes to upper secondary assessments in six countries: Adaptations and reactions. [Paper presentation]. Annual conference of the Association for Educational Assessment – Europe (AEA-Europe), Malta. https://2023.aea-europe.net/ 

Rushton, N., Vitello, S., Suto, I. (2021). What do we mean by question paper error? An analysis of criteria and working definitions. Research Matters: A Cambridge University Press & Assessment publication, 32, 67-81.

It is important to define what an error in a question paper is so that there is a common understanding and to avoid people’s own conceptions impacting upon the way in which they write or check question papers. We carried out an interview study to investigate our colleagues’ definitions of error. We found that there is no single accepted definition of a question paper error. There were three interacting aspects that participants considered when deciding whether a problem was an error: the manifestation of the error, the (potential) impact upon candidates, and the stage at which it was discovered.

• Foreword Tim Oates
• Editorial Tom Bramley
• Learning during lockdown: How socially interactive were secondary school students in England? Joanna Williamson, Irenka Suto, John Little, Chris Jellis, Matthew Carroll
• How well do we understand wellbeing? Teachers’ experiences in an extraordinary educational era Chris Jellis, Joanna Williamson, Irenka Suto
• What do we mean by question paper error? An analysis of criteria and working definitions Nicky Rushton, Sylvia Vitello, Irenka Suto
• Item response theory, computer adaptive testing and the risk of self-deception Tom Benton
• Research News Anouk Peigne

• Variations in aspects of writing in 16+ English examinations between 1980 and 2014 Gill Elliott, Sylvia Green, Filio Constantinou, Sylvia Vitello, Lucy Chambers, Nicky Rushton, Jo Ireland, Jessica Bowyer, David Beauchamp

Rushton, N. & Wilson, F. (2015). Teachers’ and employers’ views on the transition from GCSE Mathematics to A level Mathematics or employment. Research Matters: A Cambridge Assessment publication, 20, 21-27.

Mathematics is one of the core GCSE subjects, and students are required to study the subject until the end of Key Stage 4 (KS4), when they are approximately aged 16. There is no requirement for students to take a qualification in Mathematics, but almost all students do. GCSE Mathematics is important because it represents the end of students' compulsory Mathematics learning. The current study aimed to identify the areas of Mathematics that were problematic for students who had just completed GCSE Mathematics. It also aimed to discover whether there was any overlap in the skills that were considered to be problematic as preparation for A level and those considered to be problematic as preparation for employment. It uses responses from a larger survey of teachers and employers to consider three research questions: 1. What areas of Mathematics are GCSE students well/poorly prepared in? 2. What teaching is needed to bring students up to the standard for starting A level Mathematics? 3. What Mathematics training do employers run for school leavers?

Rushton, N. and Wilson, F. (2015). Teachers’ and employers’ views on the transition from GCSE Mathematics to A level Mathematics or employment. Research Matters: A Cambridge Assessment publication, 20, 21-27.

Mathematics is one of the core GCSE subjects, and students are required to study the subject until the end of Key Stage 4 (KS4), when they are approximately aged 16. There is no requirement for students to take a qualification in Mathematics, but almost all students do. GCSE Mathematics is important because it represents the end of students' compulsory Mathematics learning. The current study aimed to identify the areas of Mathematics that were problematic for students who had just completed GCSE Mathematics. It also aimed to discover whether there was any overlap in the skills that were considered to be problematic as preparation for A level and those considered to be problematic as preparation for employment. It uses responses from a larger survey of teachers and employers to consider three research questions: 1. What areas of Mathematics are GCSE students well/poorly prepared in? 2. What teaching is needed to bring students up to the standard for starting A level Mathematics? 3. What Mathematics training do employers run for school leavers?

Rushton, N. (2014). Common errors in Mathematics. Research Matters: A Cambridge Assessment publication, 17, 8-17.

When answering Mathematics questions, students often make errors leading to incorrect answers or the loss of accuracy marks. Many of these errors will be random, occurring through calculation errors or misreading of the question, and will not affect many candidates. However, some errors may be seen in a number of students’ scripts. These are sometimes referred to as common errors.

The aim of this study was to identify common errors that have been made in Mathematics exams. Three Mathematics specifications were used in this study: IGCSE Mathematics (0580), GCSE Mathematics A (J512) and GCSE Mathematics B (J567). Copies of the examiners’ reports and exam papers were obtained for all three qualifications for June 2009, 2010, 2011 and 2012. Within each examiner’s report, any common errors that candidates made were coded against a theme and sub-theme. The results were intended to inform the redevelopment of the mathematics qualifications, and to provide useful information for teachers and examiners. 

Rushton, N. (2013). Changing times, changing qualifications. Research Matters: A Cambridge Assessment publication, 16, 2-9.

During recent years there have been many changes in education and assessment in England. Since 2000, curricula have been updated, particular skills have been included then removed from assessment, several new qualifications for students in English secondary schools have been added to the Register of Regulated Qualifications and other qualifications have been withdrawn. When so many changes occur in a short space of time it is difficult to keep track of them, and the time at which they happened.

This article tracks some of the changes that have occurred in England since 2000. The article is divided into three sections: qualifications being added and withdrawn from the Register of Regulated Qualifications; changes to GCSEs (including the proposed English Baccalaureate Certificates); and changes to A levels. For each section, a time line is included to provide an overview of the most important dates alongside a summary of the major events associated with each qualification/reform.

Mehta, S., Suto, I., Elliott, G. and Rushton, N. (2013). Independent research at A level: Students’ and teachers’ experiences. Research Matters: A Cambridge Assessment publication, 15, 9-16. 

Our aims were to explore teachers’ and students’ experiences and perspectives of independent research at A level. The study focused Economics, French and Mathematics. It investigated: (i) the extent to which teachers think research and investigative skills can be developed at A level; (ii) the resources and guidance that students use; and (iii) whether subject-specific differences arise. A questionnaire and follow-on interview methodology was used. 47 Mathematics teachers, 24 Economics teachers and 15 French teachers participated. Additionally, 299 Mathematics students, 228 Economics students and 136 French students took part.

About half of the French and Economics teachers were found to assign investigative/research tasks to their students at least once a fortnight. On the other hand, about half of the Mathematics teachers set such tasks less often and a further 40% never set them at all. The frequency with which the teachers set investigation/research tasks as homework/private study showed the same subject-specific differences as the classroom context. The internet was the most frequently listed source that students across all three subjects consulted while engaging in independent research. The interview data shed further light on general and specific internet usage. Overall, the findings explain some of the variation in preparedness of new undergraduates for independent study and research-related tasks at university.

Rushton, N. (2012). The effect of scripts’ profiles upon comparability judgements. Research Matters: A Cambridge Assessment publication, 14, 10-17.

Comparability studies often involve experts’ judging students’ scripts to decide which is better. Sometimes this involves experts judging more than one paper/component from a student. Students frequently achieve a higher level in one paper/component than they do on another. The scripts from these students are described as having an uneven profile.

Uneven profile scripts have been identified as a cause of difficulty for those making judgements in comparability studies. This study investigated whether it was harder to compare uneven profiled scripts and whether uneven profiled scripts were judged more harshly.

It found that the profile of the script only affected the difficulty of making comparisons in English Literature, where the effect varied by judge. Uneven profile scripts were slightly more likely to win their comparisons, but this depended on the judge. These findings suggest that the outcome of comparability studies could be affected by the existence of uneven profile scripts.

Rushton, N., Haigh, M., and Elliott, G. (2011). The pitfalls and positives of pop comparability. Research Matters: A Cambridge Assessment publication, Special Issue 2, 52-56. 

The media debate about standards in public examinations has become an August ritual. The debate tends to be polarised with reports of ‘slipping standards’ at odds with those claiming that educational prowess has increased. Some organisations have taken matters into their own hands, and have carried out their own studies investigating this. Some of these are similar to academic papers; others are closer in nature to a media campaign. In the same way as ‘pop psychology’ is a term used to describe psychological concepts which attain popularity amongst the wider public, so ‘pop comparability’ can be used to describe the evolution of a lay-person’s view of comparability. Studies, articles or programmes which influence this wider view fall into this category and are often accessed by a much larger audience than academic papers. In this article, five of these studies are considered: Series 1 of the televised social experiment “That’ll Teach ‘em”; The Royal Society of Chemistry’s Five-Decade Challenge; the Guardian’s and the Times’ journalists (re)sitting examinations to experience their difficulty; a feature by the BBC Radio 4 programme, ‘Today’ (2009), where students discussed exam papers from 1936; and a book of O level past papers and an associated newspaper article which described students’ experiences of sitting the O level exams.

Greatorex, J., Mehta, S., Rushton, N., Hopkin, R. and Shiell, H. (2011). Developing a research tool for comparing qualifications. Research Matters: A Cambridge Assessment publication, 12, 33-42.

Comparability studies about qualification standards generally use demand or candidates’ performance as comparators. However, these can be unrepresentative for vocational and new qualifications. Consequently, other comparators need to be used. This article details the process of devising and piloting a research instrument to compare the features of cognate units from diverse qualifications and subjects.

First, knowledge was elicited from twelve experts through Kelly’s repertory grid interviews where they were asked to compare different types of qualifications. This data was analysed thematically.  Four features and several sub-features were identified. These features were used to categorise the interview data and develop the research instrument. A pilot of the instrument indicated that salient features varied between units. Therefore, the instrument is suitable for use in future comparability studies about features. However, conventions still need to be agreed for how to analyse the data that is collected using the instrument.

Greatorex, J. and Rushton, N. (2010). Is CRAS a suitable tool for comparing specification demands from vocational qualifications? Research Matters: A Cambridge Assessment publication, 10, 40-44.

The aim of the research was to ascertain whether a framework of cognitive demands, known as CRAS, is a suitable tool for comparing the demands of vocational qualifications.  CRAS was developed for use with academic examinations and may not tap into the variety of demands which vocational qualifications place on candidates.  Data were taken from a series of comparability studies by awarding bodies and the national regulator.  The data were the frameworks (often questionnaires) used to compare qualifications in these studies.  All frameworks were mapped to CRAS.  It was found that most aspects of the various frameworks mapped to an aspect of CRAS.  However, there were demands which did not map to CRAS; these were mostly affective and interpersonal demands, such as working in a team.  Affective and interpersonal domains are significant in vocational qualifications; therefore, using only CRAS to compare vocational qualifications is likely to omit key demands from the comparison.