-
Research Matters 6 Foreword
Oates, T. (2008). Foreword. Research Matters: A Cambridge Assessment publication, 6, 1.
A week in politics is a long time. In the light of this, one hundred and fifty years in assessment and qualifications is an eternity. With this timeframe, it is perhaps important for current researchers in the organisation to see themselves not as individual investigators but as both the inheritors of a long tradition of enquiry and as custodians and contributors to a continuing bequest to future generations of learners and assessment professionals.
Download
-
Research Matters 6 Editorial
Green, S. (2008). Editorial. Research Matters: A Cambridge Assessment publication, 6, 1.
In the first article Johnson explores the relationships between, and the importance of, respect, relationships and responsibility in the context of assessment related research. He shares practitioner knowledge and draws from the work of eminent researchers, particularly in the vocational field.
Download
-
'3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods?
Johnson, M. (2008). '3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods? Research Matters: A Cambridge Assessment publication, 6, 2-4.
This article focuses on the merits and challenges of using qualitative research methods, and how these can contribute positively to the study of assessment. Based on a presentation from a methods-related research seminar, this paper appeals for research engagement with those areas where assessment affects the lives of others. This appeal means not only asking the difficult questions but also having the appropriate methodologies to try to answer them. The paper goes on to champion the strengths of mixed methods approaches that allow for triangulation, complementarity (where findings gained through one method offer insights into other findings) and expansion (of the breadth and scope of the research beyond initial findings).
Download
-
Do assessors pay attention to appropriate features of student work when making assessment judgements?
Crisp, V. (2008). Do assessors pay attention to appropriate features of student work when making assessment judgements? Research Matters: A Cambridge Assessment publication, 6, 5-9.
It is via the judgements of appropriate experts that assessment decisions are made, yet the actual thought processes involved during marking or grading are under-researched. This article draws on a study of the cognitive and socially-influenced processes involved in marking and grading A level geography examinations and pilot research into the marking of GCSE coursework by teachers. This data was used to investigate whether assessors pay attention to appropriate features of student work.
Verbal protocols of assessors’ thinking aloud whilst marking and grading work were collected and measures of marker agreement were obtained. The protocols were analysed in detail using appropriate coding schemes. From the behaviours identified, a tentative model of the marking process was developed, within which features of student work affecting judgements and social and personal reactions were identified. Whilst many features that appeared to influence evaluations were clearly focussed on the criteria intended for evaluation, some were not and could have influenced evaluations. Reactions to language use or legibility (when not assessing communication), personal or emotional responses and social responses sometimes occurred before marking decisions. The article discusses whether such responses could explain variations in marks from different examiners.
Download
-
Marking essays on screen: towards an understanding of examiner assessment behaviour
Shaw, S. (2008). Marking essays on screen: towards an understanding of examiner assessment behaviour. Research Matters: A Cambridge Assessment publication, 6, 9-15.
Computer assisted assessment offers many benefits over traditional paper methods. In translating from one medium to another, however, it is crucial to ascertain the extent to which the new medium may alter the nature of the assessment and marking reliability. Appropriate validation studies must be conducted before a new approach can be implemented in high-stakes contexts. The pilot described here is the first attempt by Cambridge Assessment International Education (formerly Cambridge International Examinations) to mark on-screen extended stretches of written text for the Cambridge Checkpoint English Examination. The pilot attempts to investigate marker reliability, construct validity and whether factors such as annotation and navigation differentially influence marker performance across the on-paper and on-screen marking modes. Candidates wrote their answers on paper scripts in the normal way. The scripts were then scanned and digital images of them were sent by secure electronic link to examiners for on-screen marking at home using Scoris® software. It can be relatively hard for examiners to make a full range of annotations when marking on screen. For this reason annotation sophistication was manipulated in the pilot as well as marking mode. Four marking methods were compared: on-paper with sophisticated annotations (current practice); on-paper with simplified annotations; on-screen with sophisticated annotations; and on-screen with simplified annotations. The pilot found that paper-based and screen-based inter-examiner reliability is high for the Cambridge Checkpoint English Examination. Although inter-rater reliability is lower on-screen, it is only marginally deflated.
Download
-
Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors
Johnson, M. (2008). Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors. Research Matters: A Cambridge Assessment publication, 6, 16-19.
The assessment of a large portfolio of mainly textual evidence demands an assessor to accommodate a great deal of information. This comprehension process is influenced by the linear nature of the reading process which leads to the gradual construction of a mental representation of the text in the head of the reader (Johnson and Laird, 1983).
Understanding how assessors work with portfolios also requires us to consider how assessors integrate and combine different aspects of an holistic performance into a final judgement. Sanderson (2001) suggests that the social context of the assessor is important to consider since it recognises their participation in a community of practice (Wenger, 1998) and constitutes an ‘outer frame’ for their activity.
This study sought to explore issues of consistent assessor judgement by gathering data about individual assessors’ cognitive activity as well as the socio-contextual features in which their practices were undertaken. It focused on an OCR Nationals unit in Health and Social Care (Level 2). Six assessors were asked to ‘think aloud’ whilst they judged the unit. This commentary was then transcribed into a verbal protocol and analysed with qualitative text analysis software. A modified Kelly’s Repertory Grid (KRG) interview technique was also used to gather data about different assessors’ perceptions of constructs within the same assessment criteria.
Download
-
Annotating to comprehend: a marginalised activity?
Johnson, M. and Shaw, S. (2008). Annotating to comprehend: a marginalised activity? Research Matters: A Cambridge Assessment publication, 6, 19-24.
One of the important premises underlying this article is that the cognitive processes involved in reading can play a significant role in assessment judgements. Although we acknowledge that not all assessments of performance rely on assessors appraising written texts, many tests use written evidence as an indicator of performance. As a result, it is important to consider the role of assessors’ comprehension building when reading candidates’ textual responses, particularly where candidates are offered a greater freedom in determining the form and scope of their responses.
This paper brings together literature about linguistics and annotation practices, both empirical and theoretical, and suggests that a critical link exists between annotating and reading activities. Through making the different functions of annotation explicit the intention of this paper is to primarily amplify the importance of the impact of annotating on assessor comprehension.
Download
-
Cookery examined - 1937-2007: Evidence from examination questions of the development of a subjects over time
Elliott, G. (2008). Cookery examined - 1937-2007: Evidence from examination questions of the development of a subjects over time. Research Matters: A Cambridge Assessment publication, 6, 24-30.
This paper describes the evolution of a subject - cookery, later known as home economics and food technology - over time, as seen from the perspective of examination questions. The historical background to the examinations is explored, and examples given from examination questions through the years.
Download
-
Critical Thinking - a definition and taxonomy for Cambridge Assessment
Black, B., Chislett, J., Thompson, A., Thwaites, G. and Thwaites, J. (2008). Critical Thinking - a definition and taxonomy for Cambridge Assessment. Research Matters: A Cambridge Assessment publication, 6, 30-35.
There are a vast number of Critical Thinking definitions in the literature, which are highly varied and often multi-faceted. The construct of Critical Thinking is hotly debated, with a number of key battlegrounds. The implications of such differing conceptions reach out beyond academic journals. They impact upon educationalists in a number of practical ways, such as devising the best training or delivery model for Critical Thinking; designing and delivering valid assessments which are authentic and which nurture good Critical Thinking skills in students. The main aim of this research was to create a Cambridge Assessment definition and taxonomy for Critical Thinking.
Download
-
The future of assessment - the next 150 years?
Oates, T. (2008). The future of assessment - the next 150 years? Research Matters: A Cambridge Assessment publication, 6, 36-40.
‘Prediction is very difficult, particularly if it’s about the future’ - Niels Bohr.
This article examines the theoretical and practical issues surrounding prediction of future approaches to assessment as well as describing areas of likely development - and the form which assessment might take. It uses theory from sociology, philosophy of science and public policy analysis to examine the possibilities of prediction. It then examines emergent areas of development and interest, discussing issues, tensions and possibilities. The specific areas covered comprise: adaptive testing; authentic tasks; on demand testing; influence of technology; outcomes-based learning; teacher assessment in high accountability settings; tiered examinations; ‘levels’ and grades; and maintenance of standards and measurement error.
Download
-
Research News
The Research Division (2008). Research News. Research Matters: A Cambridge Assessment publication, 6, 42.
A summary of recent conferences and seminars, and research articles published since the last issue of Research Matters.
Download