"The scandal of poor medical research"
by Douglas G. Altman (12 July 1948 – 3 June 2018)
(Source: British Medical Journal, 308, 283—284 (1994))
by Douglas G. Altman (12 July 1948 – 3 June 2018)
(Source: British Medical Journal, 308, 283—284 (1994))
We need less research, better research, and research done for the right reasons
What should we think about a doctor who uses the wrong treatment, either wilfully or through ignorance, or who uses the right treatment wrongly (such as by giving the wrong dose of a drug)? Most people would agree that such behaviour was unprofessional, arguably unethical, and certainly unacceptable.
What, then, should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature, in both general and specialist journals, have shown that all of the above phenomena are common.1 2 3 4 5 6 7 This is surely a scandal.
When I tell friends outside medicine that many papers published in medical journals are misleading because of methodological weaknesses they are rightly shocked. Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation. Errors are so varied that a whole book on the topic,7 valuable as it is, is not comprehensive; in any case, many of those who make the errors are unlikely to read it.
Why are errors so common? Put simply, much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them. Regardless of whether a doctor intends to pursue a career in research, he or she is usually expected to carry out some research with the aim of publishing several papers. The length of a list of publications is a dubious indicator of ability to do good research; its relevance to the ability to be a good doctor is even more obscure. A common argument in favour of every doctor doing some research is that it provides useful experience and may help doctors to interpret the published research of others. Carrying out a sensible study, even on a small scale, is indeed useful, but carrying out an ill designed study in ignorance of scientific principles and getting it published surely teaches several undesirable lessons.
In many countries a research ethics committee has to approve all research involving patients. Although the Royal College of Physicians has recommended that scientific criteria are an important part of the evaluation of research proposals,8 few ethics committees in Britain include a statistician. Indeed, many ethics committees explicitly take a view of ethics that excludes scientific issues. Consequently, poor or useless studies pass such review even though they can reasonably be considered to be unethical.9
The effects of the pressure to publish may be seen most clearly in the increase in scientific fraud,10 much of which is relatively minor and is likely to escape detection. There is nothing new about the message of data or of data torture, as it has recently been called11 - Charles Babbage described its different forms as long ago as 1830.12 The temptation to behave dishonestly is surely far greater now, when all too often the main reason for a piece of research seems to be to lengthen a researcher's curriculum vitae. Bailar suggested that there may be greater danger to the public welfare from statistical dishonesty than from almost any other form of dishonesty.13
Evaluation of the scientific quality of research papers often falls to statisticians. Responsible medical journals invest considerable effort in getting papers refereed by statisticians; however, few papers are rejected solely on statistical grounds.14 Unfortunately, many journals use little or no statistical refereeing - bad papers are easy to publish.
Statistical refereeing is a form of fire fighting. The time spent refereeing medical papers (often for little or no reward) would be much better spent in education and in direct participation in research as a member of the research team. There is, though, a serious shortage of statisticians to teach and, especially, to participate in research.15 Many people think that all you need to do statistics is a computer and appropriate software. This view is wrong even for analysis, but it certainly ignores the essential consideration of study design, the foundations on which research is built. Doctors need not be experts in statistics, but they should understand the principles of sound methods of research. If they can also analyse their own data, so much the better. Amazingly, it is widely considered acceptable for medical researchers to be ignorant of statistics. Many are not ashamed (and some seem proud) to admit that they don't know anything about statistics.
The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution. Manufacturing industry has come to recognise, albeit gradually, that quality control needs to be built in from the start rather than the failures being discarded, and the same principles should inform medical research. The issue here is not one of statistics as such. Rather it is a more general failure to appreciate the basic principles underlying scientific research, coupled with the “publish or perish” climate.
As the system encourages poor research it is the system that should be changed. We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.
References
What should we think about a doctor who uses the wrong treatment, either wilfully or through ignorance, or who uses the right treatment wrongly (such as by giving the wrong dose of a drug)? Most people would agree that such behaviour was unprofessional, arguably unethical, and certainly unacceptable.
What, then, should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature, in both general and specialist journals, have shown that all of the above phenomena are common.1 2 3 4 5 6 7 This is surely a scandal.
When I tell friends outside medicine that many papers published in medical journals are misleading because of methodological weaknesses they are rightly shocked. Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation. Errors are so varied that a whole book on the topic,7 valuable as it is, is not comprehensive; in any case, many of those who make the errors are unlikely to read it.
Why are errors so common? Put simply, much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them. Regardless of whether a doctor intends to pursue a career in research, he or she is usually expected to carry out some research with the aim of publishing several papers. The length of a list of publications is a dubious indicator of ability to do good research; its relevance to the ability to be a good doctor is even more obscure. A common argument in favour of every doctor doing some research is that it provides useful experience and may help doctors to interpret the published research of others. Carrying out a sensible study, even on a small scale, is indeed useful, but carrying out an ill designed study in ignorance of scientific principles and getting it published surely teaches several undesirable lessons.
In many countries a research ethics committee has to approve all research involving patients. Although the Royal College of Physicians has recommended that scientific criteria are an important part of the evaluation of research proposals,8 few ethics committees in Britain include a statistician. Indeed, many ethics committees explicitly take a view of ethics that excludes scientific issues. Consequently, poor or useless studies pass such review even though they can reasonably be considered to be unethical.9
The effects of the pressure to publish may be seen most clearly in the increase in scientific fraud,10 much of which is relatively minor and is likely to escape detection. There is nothing new about the message of data or of data torture, as it has recently been called11 - Charles Babbage described its different forms as long ago as 1830.12 The temptation to behave dishonestly is surely far greater now, when all too often the main reason for a piece of research seems to be to lengthen a researcher's curriculum vitae. Bailar suggested that there may be greater danger to the public welfare from statistical dishonesty than from almost any other form of dishonesty.13
Evaluation of the scientific quality of research papers often falls to statisticians. Responsible medical journals invest considerable effort in getting papers refereed by statisticians; however, few papers are rejected solely on statistical grounds.14 Unfortunately, many journals use little or no statistical refereeing - bad papers are easy to publish.
Statistical refereeing is a form of fire fighting. The time spent refereeing medical papers (often for little or no reward) would be much better spent in education and in direct participation in research as a member of the research team. There is, though, a serious shortage of statisticians to teach and, especially, to participate in research.15 Many people think that all you need to do statistics is a computer and appropriate software. This view is wrong even for analysis, but it certainly ignores the essential consideration of study design, the foundations on which research is built. Doctors need not be experts in statistics, but they should understand the principles of sound methods of research. If they can also analyse their own data, so much the better. Amazingly, it is widely considered acceptable for medical researchers to be ignorant of statistics. Many are not ashamed (and some seem proud) to admit that they don't know anything about statistics.
The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution. Manufacturing industry has come to recognise, albeit gradually, that quality control needs to be built in from the start rather than the failures being discarded, and the same principles should inform medical research. The issue here is not one of statistics as such. Rather it is a more general failure to appreciate the basic principles underlying scientific research, coupled with the “publish or perish” climate.
As the system encourages poor research it is the system that should be changed. We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.
References
- Altman DG. Statistics in medical journals.Stat Med1983;1:59–71.
- Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials. A survey of three medical journals.N Engl J Med1987;317:426–32.
- Smith DG, Clemens J, Crede W, Harvey M, Gracely EJ. Impact of multiple comparisons in randomised clinical trials. Am J Med1987;83:545–50.
- Murray GD. The task of a statistical referee. Br J Surg1988;75:664–7.
- Gotzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of non-steroidal antiinflammatory drugs in rheumatoid arthritis.Controlled Clin Trials1989;10:31–59.
- Williams HC, Seed P. Inadequate size of negative clinical trials in dermatology. Br J Dermatol1993;128:317–26.
- Andersen B. Methodological errors in medical research. An incomplete catalogue.Oxford: Blackwell,1990.
- Royal College of Physicians.Guidelines on the practice of ethics committees in medical research. London: RCP,1984.
- Altman DG. Statistics and ethics in medical research. Misuse of statistics is unethical. BMJ1980;281:1182–4.
- Lock S, Wells F eds. Fraud and misconduct in scientific research. London: BMJ Publishing Group,1993.
- Mills JL. Data torturing. N Engl J Med1993;329:1196–9.
- Babbage C. Reflections on the decline of science in England. New York: Augustus M Kelley, 1970: 174–83. (Cited in Broad W, Wade N. Betrayers of the truth. Oxford: Oxford University Press, 1982: 29–30.)
- Bailar JC. Bailar's laws of data analysis. Clin Pharmacol Ther1976;20:113–20.
- Bailar JC, Mosteller FBailar JC. Communicating with a scientific audience. In: Bailar JC, Mosteller F eds.Medical uses of statistics. Waltham, MA: NEJM Books, 1986: 325–37.
- Bland JM, Altman DG, Royston JP. Statisticians in medical schools. J R Coll Physicians London 1990;24:85–6.
“Perpetually seeing bad articles in medical journals just got to me. I felt aggrieved by it. It was a waste of money, of course, and a breach of ethics, but that only occurred to me later. At the time, it just seemed wrong.”
-- Douglas G. Altman (12 July 1948 – 3 June 2018)
(Source: Remembering Professor Doug Altman, University of Oxford)
-- Douglas G. Altman (12 July 1948 – 3 June 2018)
(Source: Remembering Professor Doug Altman, University of Oxford)
"Ridiculous Career Necessity"
“The majority of statistical analyses are performed by people with an inadequate understanding of statistical methods. They are then peer reviewed by people who are generally no more knowledgeable. Sadly, much research may benefit researchers rather more than patients, especially when it is carried out primarily as a ridiculous career necessity.” (Source: "STATISTICAL REVIEWING FOR MEDICAL JOURNALS" DOUGLAS G. ALTMAN. Statics in Medicine 17, 2661—2674 (1998), p. 2662)
"Some years ago I encountered a very thought-provoking (unpublished) quotation attributed to Michael Healy:
“The majority of statistical analyses are performed by people with an inadequate understanding of statistical methods. They are then peer reviewed by people who are generally no more knowledgeable. Sadly, much research may benefit researchers rather more than patients, especially when it is carried out primarily as a ridiculous career necessity.” (Source: "STATISTICAL REVIEWING FOR MEDICAL JOURNALS" DOUGLAS G. ALTMAN. Statics in Medicine 17, 2661—2674 (1998), p. 2662)
"Some years ago I encountered a very thought-provoking (unpublished) quotation attributed to Michael Healy:
- ‘The difference between medical research and agricultural research is that medical research is done by doctors but agricultural research is not done by farmers.’
"Statistical Reform"
- "We see the impact of poor research methods everywhere around us. Reviews focusing on the appropriate use and reporting of research methods are almost universally depressing. Chalmers and Glasziou have opened our eyes to massive research waste. Ioannidis tells us that most research findings are false. Reproducibility, or lack thereof, is on many of our minds. And of course there is the late Doug Altman warning us of “the scandal of poor medical research”, way back in 1994. Had we only listened."
We are now to the point where the enemies of science, those who fear informed citizens and decision-making, are using these problems to undermine the public’s trust in our work (not that we aren’t doing a pretty good job of that ourselves). Make no mistake, science is pointless without trust, and given the serious threats we are facing as a species, it’s not a tool we want to do without." (Source: "Statistical Reform" Darren L. Dahly, Principal Statistician for the HRB Clinical Research Facility in Cork, Ireland, and a Senior Lecturer in Research Methods at University College Cork. Medium March 10, 2019)
"How scientists fool themselves – and how they can stop"
"Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts."
"Scientists rise up against statistical significance"
"Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly crucial effects."
"Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts."
- "The problem
Although the human brain and its cognitive biases have been the same for as long as we have been doing science, some important things have changed, says psychologist Brian Nosek, executive director of the non-profit Center for Open Science in Charlottesville, Virginia, which works to increase the transparency and reproducibility of scientific research. Today's academic environment is more competitive than ever. There is an emphasis on piling up publications with statistically significant results — that is, with data relationships in which a commonly used measure of statistical certainty, the p-value, is 0.05 or less. “As a researcher, I'm not trying to produce misleading results,” says Nosek. “But I do have a stake in the outcome.” And that gives the mind excellent motivation to find what it is primed to find." (Source: "How scientists fool themselves – and how they can stop" Regina Nuzzo. Nature 07 October 2015)
"Scientists rise up against statistical significance"
"Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly crucial effects."
- "Pervasive problem
Let’s be clear about what must stop: we should never conclude there is ‘no difference’ or ‘no association’ just because a P value is larger than a threshold such as 0.05 or, equivalently, because a confidence interval includes zero. Neither should we conclude that two studies conflict because one had a statistically significant result and the other did not. These errors waste research efforts and misinform policy decisions." (Source: "Scientists rise up against statistical significance" Valentin Amrhein, Sander Greenland & Blake McShane. Nature 20 MARCH 2019)