Textbooks, statistics and error are central themes of research at Cambridge Assessment [1]. So, I was intrigued when Paul Morris’ article “ Misunderstandings and omissions in textbook accounts of effect sizes ” popped through my letterbox. This is interesting in the context of research we have conducted ourselves at Cambridge Assessment.
Morris explains that for decades, scientists have over-relied on null hypothesis significance testing [2] to judge the credibility and importance of experimental results. Much scholarly work notes the limitations of the statistics used in significance testing. Many scholars suggest that a way of overcoming these limitations is to use alternative statistical approaches, such as effect sizes, which indicate the magnitude of a research finding. However, these are tricky to use, and this sparked Morris’ research interest in how textbooks explain effect sizes. He went on to analyse how effect size was explained in 138 different textbooks.
Morris found that many textbooks accurately portray some statistical information. For instance, they highlight that effect size magnitudes should only be considered as small, medium and large if there is no theoretical or practical reason for determining the importance of the effect size magnitude. Sometimes small effect sizes are important and sometimes large effect sizes are unimportant.
However, many textbooks present effect size as a simple index of the practical importance of the research result or as a measure of a real-world population effect. This obscures the reality that research study design itself influences effect size magnitude, and Morris notes that this information is shared only by a minority of textbooks.
In addition, Morris observes that often textbooks omit key debates. For example, most textbooks fail to make it explicit that different measures of effect size are appropriate for different research designs. He says that this may be because there is little consensus amongst statisticians on these issues.
Although Morris concentrates on textbooks for university students and Cambridge Assessment research focuses on school textbooks, there are connections between their research findings. Cambridge Assessment research shows that vital aspects of high quality school textbooks include “clear delineation of content – a precise focus on key concepts and knowledge” (Oates, 2014, p. 4). Unfortunately, Morris’ study suggests that many textbook descriptions of effect sizes do not have this vital feature.
This raises the question: What is the impact of omitting key debates and mis-messaging in textbooks? Morris argues that improving textbook explanations may upskill behavioural scientists. Cambridge Assessment research shows that when teachers use high quality textbooks (in a variety of subjects) this can contribute to improvements in student achievement (Oates, 2014). A common thread in work by Morris and Cambridge Assessment is that high quality textbooks play a key part in learning, and their design has educational consequences.
Jackie Greatorex, Principal Research Officer, Cambridge Assessment
[1] For example, Newton (2008), Oates (2016) and Vidal Rodeiro & Williamson (2019).
[2] A null hypothesis is a statement generally saying that there is no association or difference between two (or more) phenomena or groups of people such as; there is a similar number of male and female scientists. Significance testing is a process of rejecting or accepting the null hypothesis (statement) based on the results of particular statistical tests, including Chi Square and T tests.
______________________________________________________________________________________________________
References
Morris, P. H. (2020). Misunderstandings and omissions in textbook accounts of effect sizes, British Journal of Psychology, 111(2), 395-410, DOI:10.1111/bjop.12401
Newton, P. (2008). Recognising the error of our ways. Paper presented at the Cambridge Assessment Forum for New Developments in Educational Assessment, Downing College, Cambridge. 10 December 2008. Retrieved from https://www.cambridgeassessment.org.uk/Images/126096-dr-paul-newton-s-presentation-recognising-the-error-of-our-ways-.pdf
Oates, T. (2014). Why textbooks count. A Policy Paper. Cambridge Assessment. https://www.cambridgeassessment.org.uk/Images/181744-why-textbooks-count-tim-oates.pdf
Oates, T. (2016). The Cambridge Approach to Textbooks Principles for designing high-quality textbook and resource materials. Cambridge Assessment. https://www.cambridgeassessment.org.uk/Images/cambridge-approach-to-textbooks.pdf
Vidal Rodeiro, C. & Williamson, J. (2019). Meaningful destinations: using national data to investigate how different education pathways support young people’s progression in England, Research Papers in Education, 34(6), 725-748, DOI: 10.1080/02671522.2018.1536889