Just like Goldilocks, exam questions should be 'just right'

Just like Goldilocks, exam questions should be 'just right'

Exam question changes in the past two decades have often been made for clarity and “accessibility”, says Tim Oates. But A-level questions from decades ago now available on the web, are proving that there is still a huge appetite for demanding physics.

Social media allows global circulation of all sorts of conversations that previously would simply have been between two young people outside the exam hall — including the immediate complaint, "the questions were too hard". Indeed, at Cambridge we did some evaluation work on a set of qualifications and were struck by the students'
"Pupils in the 1980s knew a lot of physics"

responses: "…We know that the difficulty of the questions are taken into account and an A* last year is the same as an A* this year, but we don’t like papers where we know that we are not getting all the questions completely right…" Interesting stuff about contemporary views of exams. Obviously, a GCSE paper in which few of the candidates get any of the questions right is a badly designed one. If everyone is bunched at the top, there is a problem of poor measurement.

James Stigler once deliberately set an impossible maths question as part of a research programme. It showed interesting differences between a nation in which maths ability is emphasised over effort, in contrast to those in which effort is emphasised over ability. In the "ability" nation, pupils quickly abandoned attempting the question. In the "effort" nations, pupils had to be told to stop after weeks of trying. No surprises which has higher overall maths attainment and better equity in outcomes. So what questions should we have in exams, and what questions should we use to maximise learning?

All of this hints at a kind of "Goldilocks" requirement: they shouldn’t be too easy or too hard, they should be "just right". There is some evidence from the past two decades that subtle changes have
"In 1982, more than 55,000 pupils sat physics A-level; last year, about 30,000"

been introduced in exam questions, often in the name of clarity and "accessibility". These have reduced the demand of questions and papers in some key subjects, with history and physics often mentioned. This can be subtle: breaking something down into small parts, adding a diagram here, a label there. Do it for one year’s paper, and it may be a small and insignificant change. Do it for ten successive years and you might get a substantial shift. Surely all students would welcome this relaxation? However, while a less demanding exam may be welcomed in a hot exam room in May, a quiet project has been effecting a revolution in how we can look at things.

Mark Warner and Lisa Jardine-Wright, from the Cavendish Laboratory in Cambridge, have put together a team of physicists who have been going through the archives at Cambridge Assessment, examining thousands of exam questions in physics going back to the late 19th century. They have found that over the past four decades questions have been in sub-parts, reducing the requirement for candidates to break down complex problems into manageable parts. Diagrams have removed the need for them to imagine, and then represent physical systems. Small changes; but over decades, they have added up. And it is no accident that physics degrees have had to change, many moving to four-year degrees, with some considering five years.

The first response of many people is to say "…but in the 80s, not many people were taking physics A-level, so they were all of high ability." Not at all — in 1982, more
"This could be a 'bad' news story; a decline in demanding exams."

than 55,000 pupils sat physics A-level; last year, about 30,000. "Ah. But in 1980s, with those 'harder' questions, people did not have to get such a high mark to get an A…" But I have looked at the examiner reports of the time and these pupils had to get a high mark to get a high grade. They knew a lot of physics.

So this could be a "bad news" story; a decline in demanding exams. Except something very interesting has happened. The researchers have made the best historical questions available on the Isaac Physics website, in a form that keeps these demanding questions intact, and supports pupils in answering them, through prompts and hints. Excellent for learning as well as checking progress. And this is in huge demand — 10,000 hits a day at peak. Contrary to many perceptions and assumptions about the zeitgeist, there is huge appetite out there for demanding physics, a very "good news" story.

Tim Oates
Group Director of Assessment Research and Development, Cambridge Assessment

isaacphysics.org

Research Matters

Research Matters 32 promo image

Research Matters is our free biannual publication which allows us to share our assessment research, in a range of fields, with the wider assessment community.