Where is the Wisdom?
The Poverty of Medical Evidence

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:    Frankp@chiro.org

  British Medical Journal 1991 (Oct 5); 303: 798–799  

By RICHARD SMITH, Editor of BMJ

---

"Where is the wisdom we have lost in knowledge, and where," asked T. S. Eliot, "is the knowledge we have lost in information?" There are perhaps 30,000 biomedical jounals in the world, and they have grown steadily by 7% a year since the seventeenth century. [1, 2] Yet only about 15% of medical interventions are supported by, solid scientific evidence, David Eddy, professor, of health policy and management at Duke University, North Carolina, told a conference in Manchester last week.

This is partly because only 1% of the articles in medical journals are scientifically sound. [2, 3] and partly because many treatments have never been assessed at all. "If," said Professor Eddy, "it is true, as the total quality management gurus tell us, that ‘every defect is a treasure’ then we are sitting on King Solomon's mine."

What are the implications for those purchasing health care if the scientific base of medicine really is so fragile? Because, as Professor Eddy said, "it is not enough to do the thing right; it is also necessary to do the right thing." The implications for purchasers of the poverty of medical evidence were considered at the Manchester meeting, which was organised jointly by the, British Association of Medical Managers and the resource management unit of the NHS Management Executive.

Professor Eddy began his medical life as a cardiothoracic surgeon in Stanford in California but became progressively concerned about the evidence to support what he and other doctors were doing. He decided to select an example of a common condition, with well established treatments and assess in detail the evidence supporting those treatments. Beginning with glaucoma, he searched published medical reports back to 1906 and could find not one randomised controlled trial of the standard treatment. Later he traced back the confident statements in textbooks and medical journals on treating glaucoma and found that they had simply been handed down from generation to generation. The same analysis was done for other treatments, including the treatment of blockages of the femoral and popliteal arteries; the findings were similar. That experience "changed his life," and after taking a degree in mathematics at Stanford University he became a professor at Duke University and one of the consultants most in demand in the United States.

Regularly he advises those producing consensus statements, and he is suspicious of the process. The best statements are based on scientifically sound evidence, but even when it is lacking (which is usual) the statements should make clear what evidence is available. Agreement of the experienced, without evidence, is a poor basis for producing advice, and as an illustration he told the story of the consensus reached by an international group that was expert in screening for colorectal cancer.

The group, including Professor Eddy, met all over the world for three days a year for five years. At the end the group recommended a protocol based on regular faecal occult blood tests and sigmoidoscopy. Professor Eddy asked each member of the group then to make a private estimate of how much mortality would be reduced by such a policy: the answers ranged from 0 to almost 100% and were randomly distributed within that range. Yet the consensus had been unanimous. As Hippocrates said, experience is fallacious.

Professor Eddy now runs courses for expert groups trying to achieve consensus. Each time he asks the members to list the outcomes they are seeking and to rank the scientific evidence for each outcome from excellent to none and then describe the best available evidence. For 21 problems tackled so far, the evidence has been judged— by the experts —to be between poor and none for 17, and usually the best available evidence was something less than a randomised controlled trial. Often the evidence that was available contradicted current practice; thus of 17 randomised trials on giving lidocaine prophylactically in patients with chest pain, 16 showed no effect, and one showed a positive result—yet practice in the United States was to give lidocaine.

The weakness of the scientific evidence underlying medical practice is one of the causes of the wide variations that are well recognised in medical practice. Dr Hugh Sanderson, director of the Wessex Cancer Intelligence Unit, illustrated the wide variations among observers and in referral rates, admission rates, investigations, and treatment. For example, among a sample of 172 radiotherapists 48% offered palliative treatment to patients with metastasised lung cancer only if they had symptoms, whereas 52% always offered treatment. Professor Eddy used this example to illustrate how doctors could be made not just to understand intellectually the variation in practice but also to feel it: radiotherapists could be asked to write down in secret what they would do for a particular patient and the results could then be pooled and discussed. The same process can be used with any specialty.

The evidence on effectiveness is poor, but the information needed— by purchasers, for instance— to choose among different treatments is almost never available. To choose, for example, among screening programmes you need, said Professor Eddy, data on how many people would need to be screened, how many deaths might be prevented, the cost of the screening program, and the savings. Turning to the "good news" in his presentation, he illustrated how mathematical modelling could be used to analyse limited data in order to make better decisions. He asked the audience to imagine that they had $400m to spend on screening for a population of a million adults and to choose between cholesterol screening as recommended by the national cholesterol education programme and breast screening as recommended by the American Cancer Society. These are two of the best studied screening methods, and much of the information needed for making policy decisions is available.

Cholesterol screening, according to the recommended protocol, could prevent 9620 events (including 340 sudden deaths, 2760 myocardial infarctions, and other conditions, some of them only poorly defined because of imprecise data) at a cost of $449m with a saving of $l01m (net cost $348m); breast screening would save 222 lives at a cost of $296m, but with a saving of only $5m (net cost $291m). A purchaser might thus opt for cholesterol screening, but Professor Eddy set his computer to adopt a more selective method of deciding who should be treated for hypercholesterolaemia— very much in the manner described in last week's BMJ by Professor Hugh Tunstall-Pedoe (28 September, p 744). The result is that you can prevent the same number of events with consequently the same saving (which is what Professor Eddy told his computer to do) at a cost of $192m, giving a net cost of $91m. The purchaser with $400m could then have both cholesterol and breast screening with $18m change. The computer could go further and be set to achieve higher benefits at lower costs.

This is how important information can be, but one of the problems is that the information comes out of a mathematical model— and doctors feel uncomfortable with such models. The doctors who devised the national cholesterol education program responded to his computer manipulation by saying that the protocol he had devised was too complicated to use. Professor Eddy is sympathetic but points out that medicine is far too complex an activity to be conducted by human minds unaided by computers: "We've been trying that for two millennia and look where we've got to." Planes are landed better by computers than humans— especially in rough weather— and much of medicine is more complicated than landing a plane. Professor Eddy thus has a fantasy of a health room equivalent to the control room at an airport, and the health room would contain all the information needed to make decisions to improve health. Like all fantasies this must be treated with caution, and purchasers in the NHS and elsewhere are faced now with making difficult choices with grossly inadequate information.

The afternoon session of the conference tackled some of the hard questions faced by purchasers. Firstly, should they specify care processes or health gain in contracts? Whatever you go for, make your decision in an alliance with providers, said one group asked to answer the question. Professor Eddy thought that for now purchasers would have to specify processes rather than outcomes, because outcomes were delayed and probabilistic. Secondly, where should purchasers get their advice on clinical advances? They had to go to local providers, everybody agreed, but "get them to provide their evidence," said Professor Eddy. Go as well, the meeting agreed, to national and international bodies, set priorities, and look for sophisticated analyses that use the best data. And try to avoid duplication, said Professor Eddy: "In the United States we have 200 groups working on the same 15 problems." Finally, how can doctors be encouraged to use this information? There were mutterings about participation, ownership, alliances, quality assurance, education, and cultural change, but Professor Eddy concluded: "Get doctors to understand how much they need reliable information. What could be worse than two millennia spent making life and death decisions with inadequate information?"

REFERENCES:

1. de Solla Price D.
   The development and structure of the biomedical literature.
   In: Warren KS, ed. Coping with the biomedical literature.
   New York: Praegel, 1981 :3-16.
   

2. Lock S.
   lntroduction.
   In: LockS,ed. The Future of Medical Journals.
   London: BMJ 1991: 1-8
   

3. Williamson, JW, Goldschmidt PG, Colton T.
   The quality of medical literature: an analysis of validalilon assessments.
   In: Bailar JC, Mosteller F. Medical use of statistics.
   Waltham. Massachusetts: NEJM Books, 1986.

Return to SAFETY OF CHIROPRACTIC