-
Research Matters 9 - Foreword
Oates, T. (2010). Foreword. Research Matters: A Cambridge Assessment publication, 9, 1.
Once again, Research Matters is testimony to the breadth of focus and method which is present in research across Cambridge Assessment. The editorial rehearses the detail of the wide variety of approaches taken but what readers might miss is the crucial nature of the statistical reports listed at the rear of this issue. These are all available on the Cambridge Assessment main website and, in my view, make an essential contribution to the transparency of the examinations system.
Download
-
Research Matters 9 - Editorial
Green, S. (2010). Editorial. Research Matters: A Cambridge Assessment publication, 9, 1.
In this issue the themes range from the awareness and usage of non-standard English among sixteen year olds to examiner judgements in awarding
Download
-
Exploring non-standard English amongst teenagers
Black, B. (2010). Exploring non-standard English amongst teenagers. Research Matters: A Cambridge Assessment publication, 9, 2-11.
The main aim of this research was to measure the levels of awareness of non-standard English (NSE) amongst GCSE level students. Through use of a questionnaire/assessment instrument, it sought to add to the empirical research on non-standard written English in young people at GCSE level in the following areas:
- which NSE forms were most and/or least recognised;
- whether respondents could produce Standard English versions of the NSE forms;
- whether respondents could spontaneously use the term non-standard English when asked to identify the type of English used in the assessment instrument;
- perceptions of NSE;
- whether characteristics of respondents (gender, school type, region) produced any differences in recognition and production of NSE.
Download
-
The evolution of international History examinations: an analysis of History question papers for 16 year olds from 1858 to the present
Shaw, S. and Cooke, G. (2010). The evolution of international History examinations: an analysis of History question papers for 16 year olds from 1858 to the present. Research Matters: A Cambridge Assessment publication, 9, 11-19.
The focus of this article is on international History examinations for 16 year olds from 1858 to the present day and examines the historical/cultural context for, and the outworking of, the setting of these examinations in the medium of English. Specific reference points throughout this period have been taken and a linguistic analysis applied to the question papers. A variety of archive material has been used to show more general developmental changes to the curriculum throughout the period. The article examines the language used, the candidate base, the regional differences of the papers and the examiner expectations. To put these findings into context, other sources, including examination regulations, examiners’ reports and subject committee papers have also been studied.
In 1858 when the Cambridge Local Examinations were introduced, history was a compulsory element of the Junior examination. Candidates had to pass in a whole range of subjects to gain a school leaving certificate and English history could not be avoided. 150 years later there is no doubt that school examinations for 16 year olds have undergone radical transformation and for history examinations to have remained unchanged would be unthinkable. The interest lies not in the fact that the examinations changed but in the way they have changed. While the trend is inevitably towards a more familiar, contemporary style, this study also shows that the pace and particular directions of change have been of a less predictable nature.
Download
-
The reliabilities of three potential methods of capturing expert judgement in determining grade boundaries
Novakovic, N. and Suto, I. (2010). The reliabilities of three potential methods of capturing expert judgement in determining grade boundaries. Research Matters: A Cambridge Assessment publication, 9, 19-24.
In England there is a strong public expectation that qualification standards should remain constant over time. At each examination session, awarding bodies must therefore determine the grade boundaries for their examinations that equate to those of previous sessions. We investigated the reliabilities of three methods for capturing the expert judgement of professional examiners who are responsible for maintaining year-on-year examination standards. The methods were those used in: traditional (current) awarding; Thurstone pairs; and rank ordering.
In the context of setting grade boundaries in AS level Biology and GCSE English, we conducted a three-way comparison of the intra-method and inter-method reliabilities of the three methods. For each subject, three mutually exclusive sets of examination scripts were created, which were matched for mark. Three groups of ten ‘judges’ (examiners, matched for experience of the methods) made judgements using each of the three methods on a different set of scripts. It was found that for both subjects, the traditional awarding and Thurstone pairs methods generated very similar boundary marks, except for at the biology A/B grade boundary. The boundary marks generated by rank ordering were all on the lenient side for biology, whereas for the English C/D grade boundary, they were on the severe side.
Download
-
How do examiners make judgements about standards? Some insights from a qualitative analysis
Greatorex, J. (2010). How do examiners make judgements about standards? Some insights from a qualitative analysis. Research Matters: A Cambridge Assessment publication, 9, 24-32.
There is a good deal of research about how judgements are made in awarding when A level and GCSE grade boundaries are chosen. There is less research about how judgements are made in Thurstone paired comparisons and rank ordering (popular methods in comparability studies to compare grading standards). Therefore, the research question for the present study is ‘how do Principal Examiners (PEs) make judgements about standards in awarding, Thurstone paired comparisons and rank ordering?’ The present article draws from a wider project in which Principal Examiners thought aloud whilst making judgements about the quality of candidates’ work and grading standards in awarding, Thurstone paired comparisons and rank ordering situations analogous to how these methods are practised. For the present analysis a coding frame was developed to qualitatively analyse the think aloud data. The coding frame constituted codes grounded in the think aloud data and grade descriptors from the qualification specification. It was found that overall the Principal Examiners attended to valid factors such as where marks were gained, responses to key questions and characteristics of candidates’ work that were in the grade descriptors.When the importance of each factor was considered there were some similarities and some differences between the methods. Implications and recommendations are discussed.
Download
-
Statistical reports
The Statistics Team (2010). Statistical Reports. Research Matters: A Cambridge Assessment publication, 9, 39.
The ongoing Statistics Reports Series provides statistical summaries of various aspects of the English examination system, such as trends in pupil uptake and attainment, qualifications choice, subject combinations and subject provision at school. This article contains a summary of the most recent additions to this series.
Download
-
'Key discriminators' and the use of item level data in reporting
Bramley, T. (2010). 'Key discriminators' and the use of item level data in reporting. Research Matters: A Cambridge Assessment publication, 9, 32-38.
As more examination papers in general qualifications (GCSEs and A levels) are scanned and marked on screen, the marks on individual questions or question parts are collected automatically, and are referred to as item level data (ILD).The analysis of ILD is available for use in awarding meetings (where the grade boundaries are decided). This article discusses the theoretical rationale for using ILD in awarding, presents some possible formats for displaying data, and suggests ways in which the data could be used in practice.
Download
-
Research News
The Research Division (2010). Research News. Research Matters: A Cambridge Assessment publication, 9, 40.
A summary of recent conferences and seminars, and research articles published since the last issue of Research Matters.
Download