Tim Oates CBE

• Foreword Tim Oates
• Editorial Victoria Crisp
• Troubleshooting in emergency education settings: What types of strategies did schools employ during the COVID-19 pandemic and what can they tell us about schools’ adaptability, values and crisis-readiness?Filio Constantinou
• How long should a high stakes test be?Tom Benton
• Core Maths: Who takes it, what do they take it with, and does it improve performance in other subjects?Tim Gill
• Does typing or handwriting exam responses make any difference? Evidence from the literatureSanti Lestari
• Comparing music recordings using Pairwise Comparative Judgement: Exploring the judge experienceLucy Chambers, Emma Walland and Jo Ireland
• Research NewsLisa Bowett

Oates, T. (2024). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 37, 4.

The explosion of activity around large language models since the release of ChatGPT in November 2022 has pushed some very important assessment issues aside. It is vital not to forget them. The principles and criteria for high quality assessment - whether low or high stakes in character - have been carefully accumulated through experience and research. It's extremely important to remember that innovation in assessment should rest on solid foundations.

• Foreword Tim Oates
• Editorial Tom Bramley
• Extended Reality (XR) in mathematics assessment: A pedagogical visionXinyue Li
• Does ChatGPT make the grade?Jude Brady, Martina Kuvalja, Alison Rodrigues, Sarah Hughes
• How do approaches to curriculum mapping affect comparability claims? An analysis of mathematics curriculum content across two educational jurisdictionsNicky Rushton, Dominika Majewska, Stuart Shaw
• Exploring speededness in pre-reform GCSEs (2009 to 2016)Emma Walland
• A Short History of the Centre for Evaluation and Monitoring (CEM)Chris Jellis
• Research NewsLisa Bowett

Oates, T. (2023). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 36, 4.

“…Education will never be the same again, post-pandemic…” is one view. “Back to normal…” is another. Neither position captures the reality of what we experienced and what we now see. For sure, following the wide and accelerated adoption of digital learning during the pandemic we need to understand and implement best practice in hybrid learning. We must understand this massive natural experiment in educational innovation, ensuring that we can regain the progress in closing gaps which we saw prior to COVID.

• Foreword Tim Oates
• Editorial Tom Bramley
• The prevalence and relevance of Natural History assessments in the school curriculum, 1858–2000: a study of the Assessment ArchivesGillian Cooke
• The impact of GCSE maths reform on progression to mathematics post-16Carmen Vidal Rodeiro, Joanna Williamson
• An example of redeveloping checklists to support assessors who check draft exam papers for errorsSylvia Vitello, Victoria Crisp, Jo Ireland
• An analysis of the relationship between Secondary Checkpoint and IGCSE resultsTim Gill
• Synchronous hybrid teaching: how easy is it for schools to implement?Filio Constantinou
• Research NewsLisa Bowett

Oates, T. (2023). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 35, 4.

The brilliant 1999 paper “The case for ‘home internationals’ in comparative research” (Raffe et al., 1999) outlined just how rich could be the comparisons between Scotland, Northern Ireland, Wales and England. This remains salient and important. It forms a key part of the methodological background to and motivation for the four-nation study included in this issue of Research Matters, and also helps in resolving some of the policy dilemmas and conflicts raised by the “future of education” debates described in the final article here.

• Foreword Tim Oates
• Editorial Tom Bramley
• Creating Cambridge Learner Profiles: A holistic framework for teacher insights from assessments and evaluationsIrenka Suto
• A conceptual approach to validating competence frameworksSimon Child, Stuart Shaw
• Teachers’ and students’ views of access arrangements in high stakes examinationsCarmen Vidal Rodeiro, Sylwia Macinska
• Who controls what and how? A comparison of regulation and autonomy in the UK nations’ education systemsPia Kreijkes, Martin Johnson
• Assessment in England at a crossroads: which way should we go?Tony Leech
• Research NewsLisa Bowett

Oates, T. (2022). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 34, 4.

“Lest we forget.” As a nation, England in 2020 ramped up its research on young people and the formal data requirements of schools – ranging from interviews and surveys of pupils and teachers to national submission of data on pupil absences. The findings of all studies in England converge on a single view of the disruption – while a few children benefited from the processes of remote learning and time in the home, the pattern of impact for the vast majority of the children in the country is negative, highly individualised and variable. It cannot be a “return to normal” in our support for any of these affected groups; they require us not to forget them. We need to research them for the purpose of action – to understand their circumstances and individual and collective experiences, and to put in place effective evidence-based support. And quickly.

• Foreword Tim Oates
• Editorial Tom Bramley
• Learning loss in the Covid-19 pandemic: teachers’ views on the nature and extent of loss Matthew Carroll, Filio Constantinou
• Which assessment is harder? Some limits of statistical linking Tom Benton, Joanna Williamson
• Progress in the first year at school Chris Jellis
• What are "recovery curricula" and what do they include? A literature review Martin Johnson
• What's in a name? Are surnames derived from trades and occupations associated with lower GCSE scores? Joanna Williamson, Tom Bramley
• Research News Lisa Bowett

• Foreword Tim Oates
• Editorial Tom Bramley
• A summary of OCR’s pilots of the use of Comparative Judgement in setting grade boundaries Tom Benton, Tim Gill, Sarah Hughes, Tony Leech
• How do judges in Comparative Judgement exercises make their judgements? Tony Leech, Lucy Chambers
• Judges' views on pairwise Comparative Judgement and Rank Ordering as alternatives to analytical essay marking Emma Walland
• The concurrent validity of Comparative Judgement outcomes compared with marks Tim Gill
• How are standard-maintaining activities based on Comparative Judgement affected by mismarking in the script evidence? Joanna Williamson
• Moderation of non-exam assessments: is Comparative Judgement a practical alternative? Carmen Vidal Rodeiro, Lucy Chambers
• Research News Lisa Bowett

Oates, T. (2022). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 33, 4.

In this edition of Research Matters we are seeing significant refinement in both application and thinking associated with Comparative Judgement. Genuinely ground-breaking, the wide-ranging studies and projects examine its limits and processes as well as its relation to existing assessment approaches. There’s one aspect of this edition which I really commend – it not only explores the characteristics of Comparative Judgement through carefully designed empirical work, it increases our understanding of the processes of human judgement within it.

Oates, T. (2021). Foreword. Research Matters: A Cambridge University Press & Assessment publication, 32, 4.

By necessity, in response to Covid-19, nations around the world have had to undertake one of the biggest natural experiments in education which we ever have seen. And now we have a huge challenge in trying to understand what actually happened, what impact there has been for young people, schools and society, and what continuing consequences will flow through schooling, society and the economy.

• Foreword Tim Oates
• Editorial Tom Bramley
• Learning during lockdown: How socially interactive were secondary school students in England? Joanna Williamson, Irenka Suto, John Little, Chris Jellis, Matthew Carroll
• How well do we understand wellbeing? Teachers’ experiences in an extraordinary educational era Chris Jellis, Joanna Williamson, Irenka Suto
• What do we mean by question paper error? An analysis of criteria and working definitions Nicky Rushton, Sylvia Vitello, Irenka Suto
• Item response theory, computer adaptive testing and the risk of self-deception Tom Benton
• Research News Anouk Peigne

Oates, T. (2021). Foreword. Research Matters: A Cambridge Assessment publication, 31, 4.

And then came pandemic. The Research team at Cambridge Assessment turned from the established research programme not only because of disruption to work, but because the organisation needed to focus on how best to ensure continued delivery of the important public goods which assessment and qualifications supply to individuals, society and the economy.

• Foreword Tim Oates, CBE
• Editorial Tom Bramley
• Attitudes to fair assessment in the light of COVID-19 Stuart Shaw, Isabel Nisbet
• On using generosity to combat unreliability Tom Benton
• A guide to what happened with Vocational and Technical Qualifications in summer 2020 Sarah Mattey
• Early policy response to COVID-19 in education—A comparative case study of the UK countries Melissa Mouthaan, Martin Johnson, Jackie Greatorex, Tori Coleman, Sinead Fitzsimons
• Generation Covid and the impact of lockdown Gill Elliott
• Disruption to school examinations in our past Gillian Cooke, Gill Elliott
• Research News Anouk Peigne

Oates, T. (2020). Foreword. Research Matters: A Cambridge Assessment publication, 30, 1.

Although changes in life and education have been effected by the pandemic— and some of them may remain permanent—we must not forget the things which we worried about before it hit, including the curriculum and assessment matters covered in the articles here.

• Foreword Tim Oates, CBE
• Editorial Tom Bramley
• A New Cambridge Assessment Archive Collection Exploring Cambridge English Exams in Germany and England in JPLO Gillian Cooke
• Perspectives on curriculum design: comparing the spiral and the network models Jo Ireland, Melissa Mouthaan
• Context matters—Adaptation guidance for developing a local curriculum from an international curriculum framework Sinead Fitszimons, Victoria Coleman, Jackie Greatorex, Hiba Salem, Martin Johnson
• Setting and reviewing questions on-screen: issues and challenges Victoria Crisp, Stuart Shaw
• A way of using taxonomies to demonstrate that applied qualifcations and curricula cover multiple domains of knowledge Irenka Suto, Jackie Greatorex, Sylvia Vitello, Simon Child
• Research News Anouk Peigne

• Foreword Tim Oates, CBE
• Editorial Tom Bramley
• Accessibility in GCSE Science exams – Students' perspectives Victoria Crisp and Sylwia Macinska
• Using corpus linguistic tools to identify instances of low linguistic accessibility in tests David Beauchamp, Filio Constantinou
• A framework for describing comparability between alternative assessments Stuart Shaw, Victoria Crisp, Sarah Hughes
• Comparing small-sample equating with Angoff judgement for linking cut-scores on two tests Tom Bramley
• How useful is comparative judgement of item difficulty for standard maintaining? Tom Benton
• Research News Anouk Peigne

Oates, T. (2020). Foreword. Research Matters: A Cambridge Assessment publication, 29, 1.

‘Improved accessibility’ has been vigorously pursued in contemporary assessments in England, and has featured in other many nations’ discussions of fairness and bias. Perhaps it would better be described as ‘removal of sources of construct-irrelevant score variation’.

Oates, T. (2020). Editorial. Research Matters: A Cambridge Assessment publication, Special Issue 3 (First published 2012), 2.

This Special Issue of Research Matters consists of something more akin to a monograph – offering an overview of an in-depth empirical validation study. Far from being mere pragmatic empiricism, this study is driven by a complex mix of theoretical underpinning, practical validation, and ethical concern.

• Foreword Tim Oates, CBE
• Editorial Tom Bramley
• Which is better: one experienced marker or many inexperienced markers? Tom Benton
• "Learning progressions": A historical and theoretical discussion Tom Gallacher, Martin Johnson
• The impact of A Level subject choice and students' background characteristics on Higher Education participation Carmen Vidal Rodeiro
• Studying English and Mathematics at Level 2 post-16: issues and challenges Jo Ireland
• Methods used by teachers to predict final A Level grades for their students Tim Gill
• Research News David Beauchamp

Oates, T. (2019). Foreword. Research Matters: A Cambridge Assessment publication, 28, 1.

Attainment and equity; there are many things which affect each. Enhancing both without improving one at the expense of the other requires the kind of analyses we present here, and continued sophisticated public policy effort.

Bateman, A., Chakroun, B., Coles, M. & Oates, T. (2018). Discussion paper on quality assurance of qualifications in technical and vocational education and training. United Nations Educational, Scientific and Cultural Organization (UNESCO).

Oates, T. (2011). Foreword. Research Matters: A Cambridge Assessment publication, Special Issue 2, 1.

Comparability is an area beset by assumptions, trammelled by methodological dispute, and regarded, by some, as a bankrupt pursuit.This edition emphasises the need to be clear about definitions of comparability, to specify precise objectives, to be discriminating in respect of choice of method, and to understand the utility and limitations of findings. Our conclusion is that comparability is NOT a bankrupt activity. It is complex, demanding (both theoretically and practically), and frequently produces indicative rather than definitive findings. But it remains a vital part of both management of, and research on, qualifications and assessments.

Oates, T., Sykes, E., Emery, J., Bell, J.F. and Vidal Rodeiro, C.L. (2009). Research Matters: A Cambridge University Press & Assessment publication, A selection of articles (2011) 20-21. First published in Research Matters, Issue 8, June 2009

For years, evidence of a birthdate effect has stared out of qualifications data for the United Kingdom; summer-born children appear to be strongly disadvantaged. Whilst those responsible for working on this data have tried to bring public attention to this issue, it has been neglected by agencies central to education and training policy.

Researchers at Cambridge Assessment have had a long interest in the birthdate effect because it is so readily observable in the assessment data with which they have worked. More recently, Cambridge Assessment decided to review the issue with the intention to advance the understanding of the extent and causes of the birthdate effect in the English education system. Although the review focuses on understanding the birthdate effect in England, it uses international comparisons as one means of throwing light on key factors. This article outlines the findings of the review.

Bramley, T. and Oates, T. (2011). Rank ordering and paired comparisons - the way Cambridge Assessment is using them in operational and experimental work. Research Matters: A Cambridge Assessment publication, 11, 32-35.

In this article we describe the method of paired comparisons and its close relative, rank-ordering. Despite early origins, these scaling methods have been introduced into the world of assessment relatively recently, and have the potential to lead to exciting innovations in several aspects of the assessment process. Cambridge Assessment has been at the forefront of these developments and here we summarise the current ‘state of play'.

Oates, T., Sykes, E., Emery, J., Bell, J.F. and Vidal Rodeiro, C.L. (2009). Research Matters: A Cambridge Assessment publication, 8, 43-45.

For years, evidence of a birthdate effect has stared out of qualifications data for the United Kingdom; summer-born children appear to be strongly disadvantaged. Whilst those responsible for working on this data have tried to bring public attention to this issue, it has been neglected by agencies central to education and training policy.
Researchers at Cambridge Assessment have had a long interest in the birthdate effect because it is so readily observable in the assessment data with which they have worked. More recently, Cambridge Assessment decided to review the issue with the intention to advance the understanding of the extent and causes of the birthdate effect in the English education system. Although the review focuses on understanding the birthdate effect in England, it uses international comparisons as one means of throwing light on key factors. This article outlines the findings of the review.

Oates, T. (2008). The future of assessment - the next 150 years? Research Matters: A Cambridge Assessment publication, 6, 36-40.

‘Prediction is very difficult, particularly if it’s about the future’ - Niels Bohr. 

This article examines the theoretical and practical issues surrounding prediction of future approaches to assessment as well as describing areas of likely development - and the form which assessment might take. It uses theory from sociology, philosophy of science and public policy analysis to examine the possibilities of prediction. It then examines emergent areas of development and interest, discussing issues, tensions and possibilities. The specific areas covered comprise: adaptive testing; authentic tasks; on demand testing; influence of technology; outcomes-based learning; teacher assessment in high accountability settings; tiered examinations; ‘levels’ and grades; and maintenance of standards and measurement error.