-
Research Matters 2 Foreword
Lebus, S. (2006). Foreword. Research Matters: A Cambridge Assessment publication, 2, 1.
Welcome to the second issue of Research Matters, a biannual publication from Cambridge Assessment. The aim of this publication is to share assessment research in a range of fields with colleagues within Cambridge Assessment and in the wider assessment community and to comment on prominent research issues.
Download
-
Research Matters 2 Editorial
Green, S. (2006). Editorial. Research Matters: A Cambridge Assessment publication, 2, 1.
In this issue we report on a wide range of research topics from vocational assessment to new technologies. The first four articles describe research presented at conferences. These articles are followed by an evaluation of the Cambridge Assessment/Oxford University automatic marking system by Nicholas Raikes... The two articles from The Financial Times, reprinted with their permission, describe what happened when four Financial Times’ columnists agreed to sit some of the 2005 papers. In his article, ‘The Cambridge Assessment Network’, Andrew Watts outlines the way in which the Research Division and Cambridge Assessment Network work together to enhance professional development in the assessment community. And finally, ‘Research News’ includes conferencing information and details of recent publications.
Download
-
International perspectives on vocational education: What can we learn from each other?
Suto, I. and Green, S. (2006). International perspectives on vocational education: What can we learn from each other? Research Matters: A Cambridge Assessment publication, 2, 2-7.
The broad aim of Vocational Education and Training (VET) is to provide students with the technical skills and knowledge needed to enter the workforce. It exists for an extensive range of subject areas and may be delivered through many different kinds of training institutions and enterprises. Over the past four years, this huge area of education and assessment has received close scrutiny from the Government and others, as part of a broader review of education and skills among 14 to 19 year olds (Tomlinson, 2004; Department for Education and Skills, 2005). Given that this process has resulted in proposals for considerable reform of VET for this age group, we deemed it important to know more about the international context within which they are set. Who does VET affect globally, and what might we learn from the experiences of other countries? The aims of this project, therefore, were to identify and examine two main types of data: (i) on the extent of participation in VET and its associated assessment worldwide; and (ii) relating to key differences in the VET systems of different countries.
There were three stages to the project:
1. A review of the quantitative data available.
2. A review of other key literature.
3. A discussion group at an international conference.
In this report, we summarise some of the main findings from each stage. We conclude that, in general, there is a paucity of internationally comparable quantitative data relating to vocational education and training.
Download
-
A cognitive psychological exploration of the GCSE marking process
Suto, I. and Greatorex, J. (2006). A cognitive psychological exploration of the GCSE marking process. Research Matters: A Cambridge Assessment publication, 2, 7-11.
GCSEs play a crucial role in secondary education throughout England and Wales, and the process of marking them, which entails extensive human judgement, is a key determinant in the futures of many sixteen-year-olds. The aims of our study were to investigate the cognitive strategies used when marking GCSEs and to interpret them within the context of psychological theories of human judgement.
Two GCSE examinations were considered: an intermediate tier Mathematics paper, which used a ‘points-based’ marking scheme, and a foundation tier Business Studies paper, which used a ‘levels-based’ scheme. For each subject, a group of six experienced examiners marked four identical script samples each. The first three of these samples were marked silently. Whilst marking the fourth sample, the examiners were asked to ‘think aloud’ concurrently. Using a semi-structured interview schedule, the examiners were later questioned about their marking experiences retrospectively.
A qualitative analysis of the verbal protocol data enabled us to propose a tentative model of marking, which includes five distinct cognitive marking strategies: matching, scanning, evaluating, scrutinising, and no response. These strategies were broadly validated not only in the retrospective interviews with the participating examiners, but also by other senior mathematics and business studies examiners.
Download
-
Examiners annotations: practice and purpose
Crisp, V. and Johnson, M. (2006). Examiners annotations: practice and purpose. Research Matters: A Cambridge Assessment publication, 2, 11-14.
The processes of reading and writing are recognised to be inextricably intertwined. Writing helps to support cognitive demands made upon the reader whilst processing a text (e.g., O’Hara, 1996; Benson, 2001). Examiners annotate scripts whilst marking (e.g., underlining, circling, using abbreviations or making comments) and this may reflect the cognitive support for comprehension building that annotations can provide. There is also some existing evidence that annotations might act as a communicative device in relation to accountability and that annotating might have a positive influence on markers’ perceptions and affect their feelings of efficacy.
This research investigated the use of annotations during marking and the role that annotations might be playing in the marking process. Six mathematics GCSE examiners and six business studies GCSE examiners who had previously been standardised to mark the paper were recruited. Examiners initially marked ten scripts which were then reviewed by their Team Leader. Examiners then marked a further 46 (Business Studies) or 40 (Mathematics) scripts.
The examiners later attended individual meetings with researchers. The session began with each examiner marking a small number of new scripts to re-familiarise themselves with the examination paper and mark scheme. A researcher then observed each examiner as they continued marking a few further scripts. Each examiner was interviewed about their use of annotations.
The findings portray a clear sense that markers in both subjects believed that annotating performed two distinct functions. The first appeared to be justificatory, communicating the reasons for their marking decisions to others. This mirrors the statutory requirements for awarding bodies to establish transparent, accountable procedures which ensure quality, consistency, accuracy and fairness. The second purpose was to support their thinking and marking decisions. In addition to helping markers with administrative aspects of marking (for example, keeping a running tally of marks), there are claims that annotations also support higher order reading comprehension processes.
Download
-
Judging learners' work on screen: Issues of validity
Greatorex, J. (2006). Judging learners' work on screen: Issues of validity. Research Matters: A Cambridge Assessment publication, 2, 14-17.
Current developments in Cambridge Assessment and elsewhere include assessors marking digital images of examination scripts on computer, rather than the original scripts on paper, and judges marking and moderating digitally produced coursework on computer, rather than on paper. One question such innovations raise is whether marks from judgements made about the same work presented on computer and on paper are comparable. Generally the literature concerning the on-screen marking of tests and examinations suggests that on-paper and on-screen scores are indeed comparable (e.g., Bennett, 2003; Greatorex, 2004), although Fowles and Adams (2005) report that differences have been found in studies by Whetton and Newton (2002), Sturman and Kispal (2003) and Royal-Dawson (2003). Our concern in this discussion article is that even when double marking studies find high levels of agreement between marks for the same work judged in different modes, issues of validity might be masked. We are thinking of validity in terms of the cognitive processes of the assessor when reaching a judgement, and how well these reflect the judgements that were intended when the assessment was devised.
Download
-
The Cambridge Assessment/Oxford University automatic marking system: Does it work?
Raikes, N. (2006). The Cambridge Assessment/Oxford University automatic marking system: Does it work? Research Matters: A Cambridge Assessment publication, 2, 17-21.
In the first issue of Research Matters, Sukkarieh et al. (2005) introduced our work investigating the automatic marking of short, free text answers to examination questions. In this article, I give details and results of an evaluation of the final prototype automatic marking system that was developed.
Download
-
The curious case of the disappearing mathematicians
Bell, J. F. and Emery, J. (2006). The curious case of the disappearing mathematicians. Research Matters: A Cambridge Assessment publication, 2, 21-23.
This article shows that an apparently alarming fall in the number of A level mathematics students between 1989 and 2001 can be accounted for by changes in the structure of the mathematics A level leading to candidates dropping out rather than failing, and to demographic changes.
Download
-
What happens when four Financial Times' journalists go under the eye of the invigilator?
Green, M. (2006). What happens when four Financial Times' journalists go under the eye of the invigilator? Research Matters: A Cambridge Assessment publication, 2, 23-25.
This article details the experiences of four journalists taking an A level or AS level examination paper apiece as an experiment. Only a small amount of preparation was undertaken and the purpose of the experiment was to see how well they would perform based upon their existing knowledge. A detailed description of the journalists' varied emotions is provided as well as their reflections upon the content and standards of the papers in question. The article was published in The Financial Times on Saturday August 20th 2005 and is reproduced in Research Matters with their permission.
Download
-
'I expect to get an A': How the FT writers thought they would do - how they actually did
Blitz, J., Giles, C., Kellaway, L. and Lloyd, J. (2006). 'I expect to get an A': How the FT writers thought they would do - how they actually did. Research Matters: A Cambridge Assessment publication, 2, 26.
This article is a continuation of the preceding piece 'What happens when four Financial Times' journalists go under the eye of the invigilator' which describes the reactions of journalists taking an A level or AS level examination paper apiece in a subject related to their career experience - Politics, Business Studies, Economics and Media Studies. This short piece provides details of their results and the markers' comments. The article was published in The Financial Times Weekend in 2005 and is reproduced in Research Matters with their permission.
Download
-
The Cambridge Assessment Network
Watts, A. (2006). The Cambridge Assessment Network. Research Matters: A Cambridge Assessment publication, 2, 27.
In this article, Andrew Watts describes the launch of the Cambridge Assessment Network in October 2005 and discusses the burgeoning relationship between the Network and the Research Division. The article highlights practical benefits which arise from collaboration, such as the enhanced dissemmination of knowledge across the business streams of Cambridge Assessment. Philospohical advantages, including the development and promotion of a knowledge-sharing culture, are also discussed. The Network's mission to promote professional development in educational assessment is articulated in detail for readers.
Download
-
Research news
The Research Division (2006). Research News. Research Matters: A Cambridge Assessment publication, 2, 28.
A summary of recent conferences and seminars, and research articles published since the last issue of Research Matters.
Download