Martin Johnson

Martin Johnson

I joined Cambridge Assessment in 2002, and prior to that I was a primary teacher for 10 years, during which time I completed an MA in Primary Education at the University of Reading.

My work at Cambridge Assessment has focussed on a diverse set of issues. These include studies into students’ experiences and perceptions of assessment (across primary, secondary and tertiary phases of education); advisory work on international test and curriculum development at primary and secondary level; investigation into how shifts from paper-based to computer-based assessment modes influence examiner practices; and the consideration of specific issues that relate to vocational assessment and learning contexts.

I completed a PhD at the University of Cambridge looking at how examiners communicate with each other when developing their common mark scheme understanding. I’ve also recently completed two observational studies exploring how question writers develop exam questions. I am a member of the Editorial Committee of the Journal of Vocational Education and Training, and I’m also a reviewer for seven other academic journals.

In my spare time I like to travel with my family, play guitar, and listen to loud music.

Publications

2018

A review of instruments for assessing complex vocational competence

Greatorex, J., Johnson, M. & Coleman, V. (2017). A review of instruments for assessing complex vocational competence. Research Matters: A Cambridge Assessment publication, 23, 35-42.

The aim of the research was to explore the measurement qualities of checklists and Global Rating Scales [GRS] in the context of assessing complex competence. Firstly, we reviewed the literature about the affordances of human judgement and the mechanical combination of human judgements. Secondly, we reviewed examples of checklists and GRS which are used to assess complex competence in highly regarded professions. These examples served to contextualise and elucidate assessment matters. Thirdly, we compiled research evidence from the outcomes of systematic reviews which compared advantages and disadvantages of checklists and GRS. Together the evidence provides a nuanced and firm basis for conclusions. Overall, literature shows that mechanical combination can outperform the human integration of evidence when assessing complex competence, and that therefore a good use of human judgements is in making decisions about individual traits, which are then mechanically combined. The weight of evidence suggests that GRS generally achieve better reliability and validity than checklists, but that a high quality checklist is better than a poor quality GRS. The review is a reminder that including assessors in designing assessment instruments processes can helps to maximise manageability.

A question of quality: Conceptualisations of quality in the context of educational test questions

Crisp, V., Johnson, M. and Constantinou, F. (2018) A question of quality: Conceptualisations of quality in the context of educational test questions. Research in Education (ahead of print).

2017

A culture of question writing: How do question writers compose examination questions in an examination paper?
Johnson, M. and Rushton, N. (2017). Presented at the 18th annual AEA Europe conference, Prague, 9-11 November 2017.
What is effective feedback in a professional learning context? A study of how examination markers feedback to each other on their marking performance
Johnson, M. (2017). Presented at the annual conference of the British Educational Research Association, University of Sussex, Brighton, UK, 5-7 September 2017.
More like work or more like school? Insights into learning cultures from a study of skatepark users
Johnson, M. and Oates, T. (2017). Presented at the Journal of Vocational Education and Training International Conference, University of Oxford, UK, 7-9 July 2017
Multiple voices in tests: towards a macro theory of test writing
Constantinou, F., Crisp, V., and Johnson, M. (2017).  Multiple voices in tests: towards a macro theory of test writing.  Cambridge Journal of Education (ahead of print).
How do question writers compose external examination questions? Question writing as a socio-cognitive process
Johnson, M., Constantinou, F. and Crisp, V. (2017). How do question writers compose external examination questions? Question writing as a socio-cognitive process. British Educational Research Journal (BERJ). 43(4), 700-719.

2016

Making Sense of a Learning Space: How Freestyle Scooter-riders Learn in a Skate Park
Johnson, M. and Oates, T. (2016). Informal Learning Review, 140, 17-21.
The challenges of researching digital technology use: examples from an assessment context
Johnson, M. (2016). International Journal of e-Assessment, 1(2), 1-10.
Reading between the lines: exploring methods for analysing professional examiner feedback discourse
Johnson, M. (2017). Reading between the lines: Exploring Methods for Analysing Professioanl Examiner Feedback Discourse. International Journal of Research & Method in Education, 40(5), 456-470.
Feedback effectiveness in professional learning contexts
Johnson, M. (2016). Review of Education, 4(2), 195-229.
How do question writers compose examination questions? Question writing as a socio-cognitive process
Johnson, M., Constantinou, F. and Crisp, V. (2016). Paper presented at the AEA-Europe annual conference, Limassol, Cyprus, 3-5 November 2016
'Question quality': The concept of quality in the context of exam questions
Crisp, V., Johnson, M. and Constantinou, F. (2016). Paper presented at the AEA-Europe annual conference, Limassol, Cyprus,3-5 November 2016
Writing questions for examination papers: a creative process?
Constantinou, F., Crisp, V. and Johnson, M. (2016). Paper presented at the 8th Biennial Conference of the European Association for Research in Learning and Instruction (EARLI) SIG 1 - Assessment and Evaluation, Munich, Germany, 24-26 August 2016
Researching effective feedback in a professional learning context
Johnson, M. (2016). Paper presented at the 7th Nordic Conference on Cultural and Activity Research, Helsingør, Denmark, 16-18 June 2016
All in good time: Influences on team leaders’ communication choices when giving feedback to examiners

Johnson, M. (2016). All in good time: Influences on team leaders’ communication choices when giving feedback to examiners. Research Matters: A Cambridge Assessment publication, 21, 28-33.

During the standardisation and live marking phases of an examination session it is increasingly common for team leaders and examiners to interact via a digital marking environment. This environment supports a number of quality assurance functions. Team leaders can see examiners’ real time scripts and mark submissions, and they can compare examiners’ marks with preordained definitive marks on special monitoring scripts to check marking accuracy. The digital marking system also allows team leaders to give examiners feedback on their marking. This article focuses on feedback practices, using observation, interview and survey data from 22 team leaders and six examiners to explore the rationales and communication choices involved in such practices. The analyses suggest that the objective of giving feedback is to construct messages that allow examiners insights into a team leader’s thinking, and this interaction is central to the development of an examiner’s understanding of how to interpret and apply mark schemes in accordance with their team leader’s views. This article discusses how the concept of synchrony underpins the feedback practices of team leaders and examiners, ensuring that the participants develop a shared focus so that both feedback message intention and reception are aligned.

2015

Finding the common ground: Teachers' and employers' representations of English in an assessment context
Child, S., Johnson, M., Mehta, S. and Charles, A. (2015).  English in Education, 49(2), 150-166.
Reading between the lines: exploring the characteristics of feedback that support examiners’ professional knowledge building
Johnson, M. (2015) Paper presented at the British Educational Research Association (BERA) conference, Belfast, 15-17 September 2015
Articulation work: Insights into examiners' expertise from their remote feedback sessions
Johnson, M. (2015). Communication & Language at work. 1(4), 28-52
Assessment, aim and actuality: insights from teachers in England about the validity of a new language assessment model
Johnson, M., Mehta, S. and Rushton, N. (2015). Pedagogies: An International Journal. 10(2), 128-148.

2014

Assessment for Learning in International Contexts: exploring shared and divergent dimensions in teacher values and practices
Warwick P., Shaw, S. D. and Johnson, M. (2014). Assessment for Learning in International Contexts: exploring shared and divergent dimensions in teacher values and practices. The Curriculum Journal, 25(4), 1-31.
Insights into contextualised learning
Johnson, M (2014). Insights into contextualised learning: how do professional examiners construct shared understanding through feedback? E-Learning and Digital Media, 11(4), 363-378.
A case study of inter-examiner feedback from a UK context
Johnson, M (2014). A case study of inter-examiner feedback from a UK context: Mixing research methods to gain insights into situated learning interactions. Formation et pratiques d’enseignement en questions, 17, 67-88.

2013

'Seeing what they say' - Mapping the characteristics of effective remote feedback

Johnson, M. (2013) Paper presented at the European Conference on Computer Supported Cooperative Work, 21-25 September 2013.

Can you dig it?
Johnson, M and Lewis, C. (2013). Developing an approach to validly assessing diverse skills in an archaeological context. Journal of Vocational Education and Training. 65(2), 177-192
Cambridge Assessment Qualitative Research Methods Reading Group

Johnson, M. (2013). Cambridge Assessment Qualitative Research Methods Reading Group. Research Matters: A Cambridge Assessment publication, 15, 38.

This article is an update on the status of a research-based reading group that was formed to with the intention of sharing methods-related expertise within the Cambridge Assessement group. Since 2011 a series of Research Division-based reading groups have been organised. The remit of the group was initially to bring together researchers from across the Cambridge Assessment group to look at a variety of different qualitative research methods. The initiative was considered to be a useful way of sharing expertise amongst colleagues as well as being an important opportunity to raise awareness of the ways of using qualitative research methods in Cambridge Assessment’s own research.

Assessment for Learning in International Contexts (ALIC): understanding values and practices across diverse contexts

Shaw, S., Johnson, M. and Warwick, P. (2013). Assessment for Learning in International Contexts (ALIC): understanding values and practices across diverse contexts Research Matters: A Cambridge Assessment publication, 15, 17-28.

The Assessment for Learning in International Contexts (ALIC) project sought to extend knowledge around teachers’ understandings of Assessment for Learning (AfL). Using a modified version of a survey devised by James and Pedder for use with teachers in England, evidence was gathered about the assessment practices that were highly valued by teachers across international contexts (Argentina, India, Indonesia, Saudi Arabia, and Nigeria). The extent of congruence between these values and teachers’ reported classroom practices was then explored.

In very broad terms, the items most valued by the teachers in this study demonstrate the considerable value placed upon practices linked positively to formative assessment principles and strategies. Certainly it seems that teachers have a particular concern with learning more about student learning and with promoting the development of pupil agency in assessment and learning.

2012

The Assessment for Learning in International Contexts (ALIC) Research Project
Shaw, S., Johnson, M. and Warwick, P. (2012) Research Intelligence 119, 14-15
Interpreting examiners’ annotations on examination scripts: a sociocultural analysis
Johnson, M. and Shaw, S. (2012) Irish Educational Studies 31(4), 467-485
Feedback as scaffolding: Senior Examiner monitoring processes and their effects on examiner marking
Johnson, M. and Black, B. (2012) Research in Post-Compulsory Education 17(4), 391-407
Essay Marking on Screen: Factsheet 3
Cognitive workload is greater on screen
Essay Marking on Screen: Factsheet 2
Marking behaviours differ on screen
Essay Marking on Screen: Factsheet 1
Essay marking accuracy is reliable across modes
The effects of features of examination questions on the performance of students with dyslexia
Crisp, V., Johnson, M. and Novakovic, N. (2012) British Educational Research Journal, 38(5), 813-839.
What's going on? Analysing visual data to understand context-based decision-making processes
Johnson, M. and Black, B. (2012) International Journal of Research & Method in Education, 35(3), 243-250
Technologically mediated communication: methods for exploring examiners’ real-time feedback interactions
Johnson, M. and Black, B. (2012) EARLI Special Interest Group 17 (Qualitative and Quantitative Approaches to Learning and Instruction) conference on Mixed Methods in Educational Interactions, Saxion University, Deventer, September 2012
Feedback as scaffolding: Senior Examiner monitoring processes and their effects on examiner marking
Johnson, M. and Black, B. (2012) British Educational Research Association Annual Conference, University of Manchester, September 2012
A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies
Johnson, N. and Nádas, R. (2012) Educational Research and Evaluation, 18(5) 425-440
Extended essay marking on screen: is examiner marking accuracy influenced by marking mode?
Johnson, M., Hopkin, R., Shiell, H. and Bell, J.F. (2012) Educational Research and Evaluation: An International Journal on Theory and Practice, 18, 2, 107-124
Marking extended essays on screen: exploring the link between marking processes and comprehension.
Johnson, M., Hopkin, R, and Shiell, H. (2012) E-Learning and Digital Media

2011

Extended essay marking on screen: does marking mode influence marking outcomes and processes?
Johnson, M., Hopkin, R., Shiell, H. and Bell, J. F. (2011). Paper presented at the British Educational Research Association annual conference, University of London Institute of Education, September 2011.
Can you dig it?': developing an approach to validly assessing diverse skills in an archaeological context
Johnson, M. and Lewis, C. (2011). Paper presented at the Journal of Vocational Education and Training International conference, Oxford, July 2011.
Evaluating the CRAS framework: Development and recommendations

Johnson, M. and Mehta, S. (2011). Evaluating the CRAS framework: Development and recommendations. Research Matters: A Cambridge Assessment publication, 12, 27-33.

This article reviews conceptual issues surrounding comparisons of demand through a critical evaluation of the CRAS (Complexity-Resources-Abstractness-Strategy) framework (Pollitt, Hughes, Ahmed, Fisher-Hoch and Bramley, 1998).

The article outlines the origins of the CRAS framework in the scale of cognitive demand (Edwards and Dall’Alba, 1981). The characteristics of the CRAS framework are then outlined, with attention being drawn to the assumptions that underlie these characteristic features. The article culminates in a set of recommendations and guidance that are relevant for potential users of the CRAS framework.

Extended essay marking on screen: Does marking mode influence marking outcomes and processes?

Shiell, H., Johnson, M., Hopkin, R., Nadas, R. and Bell, J. (2011). Extended essay marking on screen: Does marking mode influence marking outcomes and processes? Research Matters: A Cambridge Assessment publication, 11, 2-7

Research into comparisons between how people read texts on paper and computer screen suggests that the medium in which a text is read might influence the way that a reader comprehends that text. This is because some of the reading behaviours that support comprehension building, such as seamless navigation and annotation of text, are not easily replicated on screen.

Additional research also suggests that reading long texts can be more cognitively demanding on screen, and that this extra demand can have a detrimental effect on how readers comprehend longer texts. In the context of examination marking, there might be concerns that such a mode-related effect might lead to essays being marked less accurately when marked on screen compared with when they are marked on paper.

To investigate further the potential links between marking mode and the outcomes and processes of extended essay marking, the current project replicated an earlier study (Johnson and Nádas, 2009), replacing GCSE essays with longer Advanced GCE essays. The current project considered three broad areas of enquiry, exploring mode-related influences on (i) marking outcomes, (ii) manual marking processes and (iii) cognitive marking processes.

Extended essay marking: Does the transition from paper to screen influence examiners' cognitive workload?
Johnson, M., Hopkin, R. and Shiell, H. (2011) The International Journal of e-assessment, 1, 1, 1-25

2010

Marking essays on screen: an investigation into the reliability of marking extended subjective texts
Johnson, M., Nadas, R. and Bell, J.F. (2010) British Journal of Educational Technology, 41, 5, 814-826
Towards an understanding of the impact of annotations on returned exam scripts

Johnson, M. and Shaw, S. (2010). Towards an understanding of the impact of annotations on returned exam scripts. Research Matters: A Cambridge Assessment publication, 10, 16-21.

There is little empirical study into practices around scripts returned to centres. Returned scripts often include information from examiners about the performance being assessed. As well as the total score given for the performance, additional information is carried in the form of the annotations left on the script by the marking examiner.

Examiners’ annotations have been the subject of a number of research studies (Crisp and Johnson, 2007; Johnson and Shaw, 2008; Johnson and Nádas, 2009) but as far as we know there has been no research into how this information is used by centres or candidates and whether it has any influence on future teaching and learning. This study set out to look at how teachers and students interact with examiners’ annotations  on scripts.

This study used survey and interview methods to explore:
 
1. How do teachers and centres use annotations?
2. What is the scale of such use?
3. What importance is attached to the annotations?
4. What factors might influence the interpretation of the annotations?

School Based Assessment in International Practice
Johnson, M. and Burdett, N. (2010) Problems in Modern Education, 4, 64-73
Marking essays on screen and on paper
Johnson, M., Nadas, R. and Green, S. (2010) Education Journal, 121, 39-41
Intention, interpretation and implementation: some paradoxes of Assessment for Learning across educational contexts
Johnson, M. and Burdett, N. (2010) Research in Comparative and International Education, 5, 2, 122-130

2009

Marginalised behaviour: digital annotations, spatial encoding and the implications for reading comprehension
Johnson, M. and Nadas, R. (2009) Learning, Media and Technology, 34, 4, 323-336
An exploration of the effect of pre-release examination materials on classroom practice in the UK
Johnson, M. and Crisp, V. (2009) Research in Education, 82, 47-59
An investigation into marker reliability and other qualitative aspects of on-screen essay marking
Johnson, M., Nadas, R. and Shiell, H. (2009) British Educational Research Association (BERA) Annual Conference

2008

A case of positive washback: an exploration of the effect of pre-release examination materials on classroom practice: ECER abstract
Johnson, M. & Crisp, V. (2008) European Conference on Educational Research (ECER), Gothenburg
Exploring assessor consistency in a Health and Social Care qualification using a sociocultural perspective
Johnson, M. (2008) Journal of Vocational Education and Training, 60, 2, 173-187
Grading in competence-based qualifications – is it desirable and how might it affect validity?
Johnson, M. (2008) Journal of Further and Higher Education, 32, 2, 175-184
Judging Text Presented on Screen: implications for validity
Johnson, M. and Greatorex, J. (2008) E-Learning, 5, 1, 40-50

2007

The use of annotations in examination marking: opening a window into markers’ minds
Crisp, V. and Johnson, M. (2007) British Educational Research Journal, 33(6), 943–961
Assessors’ holistic judgements about borderline performances: some influencing factors
Johnson, M. and Greatorex, J. (2007) British Educational Research Association (BERA) Annual Conference
The effects of features of GCSE questions on the performance of students with dyslexia
Crisp, V., Johnson, M. and Novakovic, N. (2007) British Educational Research Association (BERA) Annual Conference
Does the anticipation of a merit grade motivate vocational test-takers?
Johnson, M (2007) Research in Post-Compulsory Education, 12, 2, 159-179
Grading, motivation and vocational assessment
Johnson, M. (2007) The Journal of Vocational Education and Training International Conference, University of Oxford

2006

Vocational review 2002-2006: Issues of validity, reliability and accessibility
Johnson, M. (2006) British Educational Research Association (BERA) Annual Conference
A review of vocational research in the UK 2002-2006: measurement and accessibility issues.
Johnson, M. (2006) International Journal of Training Research, 4, 2, 48-71
On-line Mathematics Assessment: The Impact of Mode on Performance and Question Answering Strategies
Johnson, M. and Green, S. (2006) The Journal of Technology, Learning and Assessment, 4, 5

2005

Judging learners’ work on screen. How valid and fair are assessment judgements?
Johnson, M. and Greatorex, J. (2005) British Educational Research Association (BERA) Annual Conference
The use of annotations in examination marking: opening a window into markers' minds
Crisp, V. and Johnson, M (2005) British Educational Research Association (BERA) Annual Conference
Concepts of Difficulty: A Child's Eye View
Johnson, M. (2005) In: M. Pandis, A. Ward and Mathews, S. R. (Eds), Reading, Writing, Thinking. 199-207. Newark, DE: International Reading Association

2004

On-line assessment: the impact of mode on students’ strategies, perceptions and behaviours
Johnson, M. and Green, S. (2004) British Educational Research Association (BERA) Annual Conference
On-line assessment: the impact of mode on student performance
Johnson, M. and Green, S. (2004) British Educational Research Association (BERA) Annual Conference

2003

Concepts of difficulty - a child's eye view
Johnson, M. (2003) British Educational Research Association (BERA) Annual Conference
Changes in Key Stage Two Writing from 1995 to 2002
Green, S., Johnson, M., O’Donovan, N. and Sutton, P. (2003) United Kingdom Reading Association Conference

2002

What makes a good writing stimulus?
Johnson, M. (2002) British Educational Research Association (BERA) Annual Conference
Comparability Study of Pupils' Writing from Different Key Stages
Green, S., Pollitt, A., Johnson, M. and Sutton, P. (2002) British Educational Research Association (BERA) Annual Conference

Research Matters

Research Matters

Research Matters is our free biannual publication which allows us to share our assessment research, in a range of fields, with the wider assessment community.