-
Research Matters 11 - Foreword
Oates, T. (2011). Foreword. Research Matters: A Cambridge Assessment publication, 11, 1.
Research Matters again establishes the importance of a highly elaborated research enterprise around assessment and qualifications.
Download
-
Research Matters 11 - Editorial
Green, S. (2011). Editorial. Research Matters: A Cambridge Assessment publication, 11, 1.
The articles in this issue illustrate the challenges facing the assessment community across a range of areas.
Download
-
Extended essay marking on screen: Does marking mode influence marking outcomes and processes?
Shiell, H., Johnson, M., Hopkin, R., Nadas, R. and Bell, J. (2011). Extended essay marking on screen: Does marking mode influence marking outcomes and processes? Research Matters: A Cambridge Assessment publication, 11, 2-7
Research into comparisons between how people read texts on paper and computer screen suggests that the medium in which a text is read might influence the way that a reader comprehends that text. This is because some of the reading behaviours that support comprehension building, such as seamless navigation and annotation of text, are not easily replicated on screen.
Additional research also suggests that reading long texts can be more cognitively demanding on screen, and that this extra demand can have a detrimental effect on how readers comprehend longer texts. In the context of examination marking, there might be concerns that such a mode-related effect might lead to essays being marked less accurately when marked on screen compared with when they are marked on paper.
To investigate further the potential links between marking mode and the outcomes and processes of extended essay marking, the current project replicated an earlier study (Johnson and Nádas, 2009), replacing GCSE essays with longer Advanced GCE essays. The current project considered three broad areas of enquiry, exploring mode-related influences on (i) marking outcomes, (ii) manual marking processes and (iii) cognitive marking processes.
Download
-
The effects of GCSE modularisation: a comparison between modular and linear examinations in secondary education
Vidal Rodeiro, C. L. and Nadas, R. (2011). The effects of GCSE modularisation: a comparison between modular and linear examinations in secondary education. Research Matters: A Cambridge Assessment publication, 11, 7-13.
As part of a recent reform in education, the assessment of GCSEs is now organised into modules which can either be taken at the end of the course in a linear fashion, or can be taken throughout the course in a modular approach to teaching and learning.
Previous research suggested that students who are assessed early in the course are disadvantaged by their immaturity, if not by their narrower experience of the subject and perform worse than those assessed at the end. Also, it has been argued that regular feedback on performance helps to identify learning needs and encourages students to do better.
The present study set out to investigate the above claims combining quantitative and qualitative research. In the quantitative strand of the research, the performances of English and mathematics students were analysed. In the qualitative strand, questionnaires and interviews with students and teachers of both subjects were conducted.
Download
-
Tracing the evolution of validity in educational measurement: past issues and contemporary challenges
Shaw, S. and Crisp, V. (2011). Tracing the evolution of validity in educational measurement: past issues and contemporary challenges. Research Matters: A Cambridge Assessment publication, 11, 14-17.
Validity is not a simple concept in the context of educational measurement. Measuring the traits or attributes that a student has learnt during a course is not like measuring an objective property such as length or weight; measuring educational achievement is less direct. Yet, educational outcomes can have high stakes in terms of consequences (e.g., affecting access to further education), thus the validity of assessments is highly important.
Tracing this trajectory of evolution, particularly through key documents such as the validity/validation chapter in editions of Educational Measurement (Cureton, 1951; Cronbach, 1971; Messick, 1989; Kane, 2006) and the Standards of Educational and Psychological Testing (AERA, APA and NCME, 1954/1955, 1966, 1974, 1985, 1999) has been important to us as part of work to develop an approach to validation for general assessments.
The concept of validity is not a new one. Conceptualisations of validity are apparent in the literature from around the turn of the twentieth century, and since that time, they have evolved significantly. Earliest perceptions of validity were that of a static property captured by a single statistic, usually an index of the correlation of test scores with some criterion (Binet, 1905; Pearson, 1896; Binet and Henri, 1899; Spearman, 1904). Through various re-conceptualisations, contemporary validity theory generally sees validity as about the appropriateness of the inferences and uses made from assessment outcomes, including some consideration of the consequences of test score use. This article traces the progress and changes in the theorisation of validity over time and the issues that led to these changes.
Download
-
Does doing a critical thinking A level confer any advantage for candidates in their performance on other A levels?
Black, B. and Gill, T. (2011). Does doing a critical thinking A level confer any advantage for candidates in their performance on other A levels? Research Matters: A Cambridge Assessment publication, 11, 20-24.
Critical Thinking AS level was introduced in schools in 2001. There is much research that shows that the teaching of Critical Thinking (CT) does improve critical thinking skills. However, there is less research which shows whether CT skills can be profitably transferred to other subject domains. The purpose of this research was to investigate whether taking an AS level in CT improves performance in other A levels. Using national examination data from 2005 and 2006 we compared the performance of CT students with those not taking CT. The results of both a basic comparison of mean A level performance and a regression analysis (which accounts for prior attainment) showed a significant and positive effect of taking CT, compared with not doing so. According to the regression results, the difference was equivalent to about one tenth of a grade per A level, although the difference was greater for those achieving a higher CT grade.
Download
-
Comparing the demand of syllabus content in the context of vocational qualifications: literature, theory and method
Novakovic, N. and Greatorex, J. (2011). Comparing the demand of syllabus content in the context of vocational qualifications: literature, theory and method. Research Matters: A Cambridge Assessment publication, 11, 25-32.
Our literature review considers the methods used in studies comparing the demands of vocational syllabus content in England. Generally, categories of demands are either derived from subject experts’ views or devised by researchers. Subsequently, subject experts rate each syllabus on each demand category and comparisons can be made. However, problems with the methods include: i) Some studies over-focus on the cognitive domain rather than the affective, interpersonal and psychomotor domains; ii) Experts vary in their interpretations of rating scales. Therefore, we suggest creating a framework of demands which includes all four domains, based on a variety of subject experts’ views of demands. The subject experts might rank each syllabus on each type of demand, thus avoiding the problem(s) of rating scales, and facilitating comparisons between syllabuses.
Download
-
Rank ordering and paired comparisons - the way Cambridge Assessment is using them in operational and experimental work
Bramley, T. and Oates, T. (2011). Rank ordering and paired comparisons - the way Cambridge Assessment is using them in operational and experimental work. Research Matters: A Cambridge Assessment publication, 11, 32-35.
In this article we describe the method of paired comparisons and its close relative, rank-ordering. Despite early origins, these scaling methods have been introduced into the world of assessment relatively recently, and have the potential to lead to exciting innovations in several aspects of the assessment process. Cambridge Assessment has been at the forefront of these developments and here we summarise the current ‘state of play'.
Download
-
A better approach to regulating qualification standards
Steinberg, B. and Hyder, S. (2011). A better approach to regulating qualification standards. Research Matters: A Cambridge Assessment publication, 11, 35-38.
Responding to the Government’s 2010 Education White Paper, this article set out four ways in which it was thought standards could best be maintained. First, users need to take the major role in specifying the content criteria of qualifications – enabling them to help set the standards. Secondly, exam boards need to agree between themselves on design criteria – enabling them to set and maintain the standard in relation to each other. Thirdly, ‘communities of practice’ need to be set up around each qualification – enabling the standards of each qualification to be owned and maintained by all those with a direct interest in them. Lastly, the regulator must focus on standards alone rather than its other current objectives. Its role in this system would be to underpin inter-exam board agreements as well as those between boards and users.
Download
-
Statistical Reports
The Research Division (2011). Statistical Reports. Research Matters: A Cambridge Assessment publication, 11, 38.
The ongoing Statistics Reports Series provides statistical summaries of various aspects of the English examination system, such as trends in pupil uptake and attainment, qualifications choice, subject combinations and subject provision at school. This article contains a summary of the most recent additions to this series.
Download
-
Research News
The Research Division (2011). Research News. Research Matters: A Cambridge Assessment publication, 11, 39-40.
A summary of recent conferences and seminars, and research articles published since the last issue of Research Matters.
Download