Multiple-choice questions (MCQs) are a widely used tool in language assessment, but designing them effectively requires careful consideration. From selecting the right distractors to ensuring questions align with language proficiency levels, crafting high-quality MCQs is both a challenge and a skill.
In this blog, Margaret Cooze, an expert in language assessment and consultant for The Assessment Network, explores the key principles behind well-constructed multiple-choice questions.
She discusses common mistakes, the role of the CEFR framework, the emergence of AI, and how her upcoming workshop series will help educators refine their approach to MCQ design for language learners.
Margaret, your workshop series is titled: MCQs for languages: Getting them right for your learners. What are some common mistakes educators make when designing MCQs for language assessment?
Multiple-choice questions are difficult to write, or I should say, they are difficult to write well! That’s true for all subjects I think, but in languages the questions usually need to be based on well-structured text which will support the keys and the distractors.
Even if the text is at sentence level, and just supports one short item, we have to think about context, topic and whether these are universally understood by all learners. All of the principles of producing good MCQs apply to languages of course, but we have to consider the language level too – making sure that we are testing the language in the text, and not the language in the questions. If learners can’t understand the questions, they can’t show us their understanding of the text! We often focus on the text and forget this. We need to consider the two elements together and it’s easy to let one lead the other.
What makes a good distractor in an MCQ, and does it change when focusing on language assessment?
Thinking about this, it’s probably the same as with other subjects to a large degree. Distractors need to be plausible and tempting for some learners, but clearly wrong. Science item writers look for typical errors that learners make when producing distractors and we do something very similar in writing language items.
Maybe for languages it’s more important to make sure we have coherent sets of options. For instance, if you’re testing a verb form, all of your options need to be verb forms, but when testing a dependent preposition, you’ll need three plausible alternatives to the key.
That’s presuming we are thinking of 4-option multiple-choice items of course – and the number of options is something I touch on in the workshops. Finding suitable distractors that form a set, which are at the right CEFR level can be hard work. I usually find I spend more time working on the distractors than on the key but it’s satisfying when you identify a good set. The distractors contribute to the difficulty of each multiple choice question so it’s vital that we get these right even if it is time-consuming!
How does The Common European Framework of Reference for Languages (CEFR) influence the design of MCQs for different language proficiency levels?
The CEFR is such a rich source of information for us to draw on in writing to different levels. All language teachers and item writers are familiar with the headline statements describing the six CEFR levels, from A1 up to C2, but there is a wealth of material on the CEFR website to break this down further.
For example, there is a searchable database of Can-do statements that help writers to check they are testing a suitable function of language for the level, or give inspiration when you’ve run out of ideas. These functions might lead you to design multiple choice tasks differently according to level. At A1 or A2 for example, we are likely to focus on smaller chunks of language, while at higher levels we’d expect learners to be able to answer questions about, for example, a change in opinion of the writer through the text.
There are some great online tools to use for checking the language level of anything you write. I don’t take them as the final word, and it's important to trust your own knowledge of what you’d expect a learner at a specific level to be able to understand, but they are really useful. I've included information about these in the workshops.
What inspired you to create this workshop on multiple-choice questions for language in assessment?
There is a lot of useful material about creating multiple choice questions out there, and the workshops delivered by The Assessment Network are excellent. However, each time I hear someone talk about ‘selecting a suitable learning objective’ or ‘testing a particular area of the syllabus’, I hear myself saying ‘but that doesn’t work for languages’!
It’s fine for knowledge-based subjects like maths and sciences, but where we are assessing skills, you need a different approach. The text we use to base multiple choice questions on, whether that is assessing skills in reading or listening, or assessing grammar and lexis, are vital to the success of a multiple-choice task.
When I started writing multiple-choice tasks I wasted a lot of time with texts that I thought were full of suitable content and language to be tested, but then found the text didn’t support the number of items I needed, or the content was too similar. I find working on text and questions side-by-side is the most successful approach for me.
With AI and digital tools evolving, how do you see technology influencing the future of language assessment and MCQ design?
AI is moving so fast and it’s undoubtedly an area which is going to have a big impact on assessment across all subjects.
Interestingly many AI tools now claim to have an understanding of the CEFR and there is evidence that there is some accuracy on this, but texts generally need more work to fine-tune them. At the moment to produce suitable text and questions you have to interact with the AI platform with ‘progressive-hint prompting’ to get the task to a state that suits your needs. In my experience, it seems to be more successful with reading texts but struggles with suitable listening texts - the features of spoken discourse aren’t as well produced.
Another area that needs work is that AI appears to produce texts which can show bias – something we work hard to avoid as item writers. So at the moment, I think it has great potential but isn’t going to put item writers out of work. In fact, skilled assessment practitioners will be needed more than ever to maintain principled assessment standards.
I’m sure I’d have much more to say on this in just a couple of years – things are moving so fast!
You can learn more about Cambridge’s approach to AI and language assessment in this latest piece of research.
Ready to enhance your multiple-choice question design for language assessment? Join Margaret’s upcoming workshop series to refine your skills and create more effective MCQs.
The Assessment Network is part of Cambridge University Press & Assessment, providing professional development for impactful assessment practices.