Publication of Peer-Reviewed CAM Research Results
Wednesday, May 16, 2001

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:    Frankp@chiro.org

DR. GORDON:   Good morning, all, and thank you for being here with us.  We would like to begin the panel with Edward Campion, please.             Presenter: Edward Campion, M.D.

DR. CAMPION:   Thank you, and thank you for the invitation to appear before the Commission.

I am an internist trained in rheumatology and geriatric medicine.  I have appointments at Massachusetts General Hospital and Harvard Medical School, and for 13 years, I have been a deputy editor at the New England Journal of Medicine.

The first commitment of a medical scientific journal is to publish research that is accurate and relevant to the journals' readers.  Hence, the journals must assess both the evidence in a report and the process by which it was obtained.  Peer review is not infallible, but it does improve the quality of scientific articles and provides editors useful information as they make decisions about manuscripts.

Some of the questions that editors at journals ask are, how important is this research question; what exactly is the question trying to be answered; how rigorously was the work conducted; are the results reproducible, generalizable; were the data analyzed in a way that is valid, statistically; does the report recognize other research that has been done on this topic.

A fundamental obligation of every type of medicine is to identify what treatments can help patients and describe the basis of that information in a way that is objective and useful.  Researchers must identify specific questions, and researchers must recognize that their beliefs and prior impressions may turn out to be mistaken.  In any controversial area, it is useful to gather basic descriptive information about what current practices, beliefs, and economic consequences, actually are.

Clinical studies must be conducted in ways that minimize bias, including biases that may come from either skeptics or enthusiasts, from investigators or patients.  Comparison with appropriate controls can increase confidence and results, as will comparisons with other treatments.

Research evaluations should try to identify what degrees of change are clinically meaningful, and recognize that small differences may not be of any real benefit to patients.  Particularly, in chronic symptomatic conditions, the natural history may include large variations, independent of any treatment.  In all clinical studies, more complete follow-up increases the strength of the evidence.

We must also recognize that some very important aspects of medical care are never captured very well by any research.  These include a patient's hopes and expectations, and inner beliefs, the provider's attitude, and manner toward the patient, the past histories of both, and the personal relationship between them.

For policy recommendations, I will make three.  First, give priority to investigation of those few types of unconventional treatment that can cause major harm, especially unregulated use of drugs, herbal products, and so-called dietary supplements.  At our journal, I should say we see little evidence of reports of rigorous research of CAM.  We have published some reports of major, even life-threatening toxicities from substances such as astrolochia, PC-Spez, and a dietary supplement, ephedra.

I believe the FDA should have responsibility for safety testing of herbal medicines and dietary supplements.

The second recommendation.  Encourage and support objective research studies, but only for modalities with evidence of genuine promise.  Beware of a research trap.  Research dollars are precious.  They should not be wasted on trying to define 1,000 ill-defined, unconventional practices, most of which are harmless.

Finally, there should be more investigation of the reasons why consumers seek out unconventional therapies.  One reasons may be dissatisfaction with our nation's conventional health care delivery system.  Thank you

DR. GORDON:  Thank you very much.

Jackie Wootton.            Presenter: Jackie Wootton, M.Ed.

MS. WOOTTON:   Jackie Wootton, Executive Editor of the Journal of Alternative and Complementary Medicine.  It is often known as the "Blue One".

I note that three of the Commissioners are on our editorial board.  One is a former member of our editorial board.  Two of the panelists today are on our editorial board, and two of the panelists today are authors in this latest issue of the journal.

We believe that the review process for the Journal of Alternative and Complementary Medicine is essentially the same as that for any major biomedical research journal.  I joined the journal in March 1996, and I am responsible, with the editor-in-chief, Kim Jobst, M.D., for running the peer-review process.

I was determined from the start to emulate the high standards of expert and impartial review that are accepted practice for all quality academic publications.  The journal was accepted for indexing by MEDLINE within its first three years of publication, back-dated to the start year, and has steadily gained a strong reputation as a serious CAM research publication.

Ideally, the criteria for accepting articles is that they are clear and coherent and scholarly, the description of methods and materials is complete and replicable, that studies are contextualized within the appropriate existing bodies of knowledge, and that claims and conclusions are supported by evidence.

When selecting reviewers, I take into consideration the author's recommendations and requests for excluding certain reviewers.  We have a large editorial board who review and are available for consultation.  I have also built up an extensive database of expert specialist reviewers for the field.

Review is anonymous, but it is not double-blinded.  That is, the reviewers know who the authors are, but the authors do not know the identity of the reviewers.  There are usually three reviewers, one with the appropriate CAM knowledge and expertise, one conventional biomedical researcher, and I am fortunate that my office in Bethesda is right next door to the NIH, so I have many friends and colleagues, and I can sort of pop across and say, hey, I need a hepatologist or whatever.  This is a very, very important part of our review process.

Then we also seek a methodologist or a statistician, and I have to admit that that is actually very difficult to sustain.  I collate the reviews and consult with my colleagues.  The editor-in-chief, Kim Jobst, ratifies the final decisions.  We require structured abstracts for the research papers, in accordance with the Cochrane review process.

Like all the major journals, we receive promotional articles sponsored by companies who need third party literature.  Like all the major journals, we request a Declaration of Interest when the manufacturing company is involved.

As for all the conventional medical journals, occasional weak papers slip through.  I am pleased to hear that you said the same from the New England Medicine.  No peer-review process is infallible.  As we all know, publication is not the end of the scientific process, fortunately.

The main reasons for rejection of papers is, weak, unsubstantiated claims, strong unsubstantiated claims, to use a variant of the quote, "Extraordinary claims demand extraordinary evidence."  Trivial, overblown papers, promotional papers, and what I label suspension of disbelief.  These are papers where the authors are so convinced by what they believe is true, that they are incapable of objectivity.

These are problems facing all journal editors, what I see as the unique challenges facing CAM journals, and our journal specifically is, first of all, paucity of material.  I receive approximately 110 submissions of research review papers in a year.

Secondly, many researchers have not been exposed to stringent scientific standards.  This point keeps being made.  There is a lack of experienced researchers who have CAM experience and insights.  For the same reason, it is hard to find reviewers who understand the code of expert and impartial review.  Another problem is the non-English speaking authors who have difficulty expressing their ideas clearly.

From unique challenges come unique opportunities.  Dr. Jobst and I, and members of the Board, spend a lot of time educating and encouraging authors.  Rather than reject out of hand, we help authors to modify their claims.  For example, we may require them to add statements on the limitations of their experiment or methodology, or to rework the paper.

An another instance, a poorly designed clinical trial may have potentially interesting preliminary data or insights that can be converted into a promising case study or a short report.  Unlike some of conventional journals that might throw such papers out, we actually try to seek out these little gems.

We tolerate successive stages of review and revision.  Most papers go through one major revision stage, but some require several iterations, so that some papers can take up to two years to get through the review process.  Our rejection rate is low, around 30 percent, as a result, to a large extent, of our educative policy.  The attrition rate is high.  Many authors don't return a revised paper for reconsideration.

We are open to negative results.  I think the field actually needs some clear negatives, as well as positives, to enhance credibility.  However, authors do not want to publish negative results.  We do have safety studies, and in this most recent issue of the journal, we have a paper from two naturopathic doctors on the safety of low-dose lariatridentarta [ph], a retrospective clinical study.

Although the underlying principles of publication are the same for CAM as for conventional medical journals, as you say, the sea, the process does have differences.  CAM articles published by conventional journals are predominantly based on the standard pharmaceutical drug model of isolating and testing single agents or interventions, typically using the randomized, controlled trial.

At the Journal of Alternative and Complementary Medicine, we encourage randomized, controlled trials, but we encourage a broader perspective, too.  Our feeling is that new theories demand innovative methods, and, provided researchers adhere to the basic scientific principles of external and internal validity, they should be free to innovate, develop new appropriate methodologies and carefully crafted techniques.



Academy for Educational Development
1825 Connecticut Avenue, N.W.
Academy Hall, 8th Floor
Washington, D.C.

Dr. Angell, in a previous session, made the distinction between the scientific principles underlying the methodology and scientific methodologies.  I think this is very important.  We can say with confidence that we understand the underlying principles, but I don't think we can say that there aren't new methods out there that we need to explore.

Unfortunately, the authors rarely do provide us with these innovative methods, and this brings me to my recommendations to the Commission.  There is a need for more academic departments and research positions, at post-doctoral level, to attract the brightest and best into the field.

There is also a need for more research money for practitioner training schools, as they are the ones with the knowledge and insights needed to develop appropriate methodologies.  There is a need to expand the range of acceptable models of quality clinical research, beyond the formulaic, large clinical trial protocols, while retaining integrity and authority, and adhering to those principles previously mentioned.

There is also a particular need for sponsorship from independent funding sources that will encourage innovative and imaginative but rigorous research.

I was speaking to a colleague a couple of days ago.  He is from Vienna.  Just as an example of what I am talking about, there are a lot of interesting groups in Europe who are doing all sorts of experimental science and trials of different methodologies.  This one group was testing out a diagnostic treatment device, rather like a vega device for testing of allergies.

You plug this machine in, and you put in various substances to which people might be allergic, and then apply part of this diagnostic technique to the patients.  Then one of them said, hey, what happens if you unplug it?  So they unplugged it, and it still worked.  Then the next question, of course, was, well, how far away does it have to be and still work, and they found that it could be as far as five feet away and it still worked.

Well, what I am saying here is, you do need people to ask these simple but dramatic questions that raise all sorts of other interesting issues.

DR. GORDON:  Time is up.

MS. WOOTTON:  Yes.  I have left the statistics.  I don't know whether you want me to say those very, very quickly.

When the journal changed from quarterly to bi-monthly at the beginning of '99, more libraries subscribed.  Currently, 60 percent of our subscriptions are medical or biomedical school libraries, and libraries in research institutions.  The other 40 percent of individual subscribers are equally divided between clinicians or practitioners, researchers, and others.  Thank you.

DR. GORDON:  Thank you.

Phil Fontanarosa.          Presenter: Phil B. Fontanarosa, M.D.

DR. FONTANAROSA:  Dr. Gordon, distinguished members of the White House Commission, thank you very much for the opportunity to participate in this meeting and to provide information on the topic of publication of complementary and alternative medicine, CAM, research results and peer-reviewed medical journals.

My remarks are intended to address several of the questions you provided by discussing, briefly, three questions relevant to peer-reviewed general medical journals, and from the perspective of my role as executive deputy editor of JAMA.

First: Why should medical journals consider publishing research on complementary and alternative medicine?

Well, as you all well know, all too well, of course, the use of complementary and alternative medicine and visits to alternative medicine practitioners have increased dramatically, along with marked increases in expenditures for these therapies and increasingly prevalent third party reimbursement for these therapies.

At academic medical centers, there has been increased interest among medical students, residents, and faculty members in gaining knowledge about alternative therapies.  Now, educational opportunities for CAM have been added to the curriculum in many medical schools.

In addition, there have been significant increases in federal funding for national academic centers dedicated to CAM research, along with increased funding from government and from industry to examine CAM therapies.

Yet, despite the increasing interest in CAM, high-quality scientific evidence that clearly establishes the safety and efficacy, or a lack thereof, for many of the widely used CAM interventions is lacking.  Consequently, physicians have traditionally viewed CAM in general, and most practices contained therein, with skepticism and a degree of mistrust, I would say.  Nonetheless, many patients use CAM therapies along with their conventional medical therapies.

I believe that physicians should be the health professionals who are aware of and responsible for coordinating all medical care for their patients.  Accordingly, physicians require reliable information on CAM to understand the complex interplay of CAM therapies and conventional therapies to be prepared to serve as resources for their patients, and to be comfortable answering their patients' questions about CAM, and most important, to provide evidence-based guidance and advice about various CAM therapies.

Ideally, this information should be based on critically evaluated methodologically rigorous and scientifically valid articles published in reputable journals, rather than on anecdotal reports, unfounded opinion, or unproven theories regarding CAM therapies.

To be relevant to readers, medical journals must respond to trends in medical science, react to trends among the public regarding medical therapies, and also must serve the needs of their readers.  In a 1997 survey, physician readers of JAMA ranked alternative medicine as the seventh most important topic of 70 topics we presented them with for the journal to address.  The Editorial Board and editorial staff, after much debate and deliberation, ranked alternative medicine in the top three subjects to address for that year.

Thus, JAMA and the nine Archives journals published by the AMA, The Archives of Internal Medicine, Archives of Pediatrics and Adolescent Medicine, and so on, planned and developed an initiative to objectively assess various CAM therapies.  We published coordinated theme issues, largely devoted to CAM topics, November of 1998.

Through this initiative, we provided more than 80 articles on CAM topics to the peer-reviewed biomedical literature.  I was responsible for planning, coordinating, and editing this issue of JAMA that we published on complementary and alternative medicine, and also helped to coordinate efforts with the Archives journals.

The second issue: How do peer-reviewed journals evaluate research on complementary and alternative medicine?

I believe peer-reviewed journals should be willing to consider papers on virtually any topic relevant to the practice of medicine, medical research, and relevant to their physician readers.  Most research papers on CAM should not be rejected out of hand, simply based on the topic, but should receive an unbiased, objective editorial evaluation based on assessment of the following: the scientific importance of the study; the methodologic rigor with which the research is conducted; the novel contribution to the existing literature; and the relevance of the study results for the journal's readers.

Evaluation of the scientific qualities should involve the same rigorous and critical appraisal used to assess studies reporting research on conventional therapies and the accepted framework of scientific evidence should and must prevail.

Some key aspects in the evaluation of CAM research studies include assessment of whether the paper presents the following: and explicit, focused research question; scientific or biologic plausibility; defined target disease or condition; established and accepted rigorous research methods, appropriate controls, for example; measurable, objectively assessed endpoints, such as by blinded assessment; meaningful patient-centered outcomes; information on adverse effects and safety data; and appropriate discussion of the clinical context, public health importance, and study limitations.

The assessment of CAM research studies frequently involves peer review.  Peer review of these studies involves selection of appropriate peer reviewers who have expertise in the research methods and subject matter of the study.  The goal is to obtain an objective and unbiased evaluation of the research.

Reviewers are identified and selected in several ways, but most commonly from a large database of reviewers we have at JAMA.  This database includes self-designated areas of expertise of the reviewers, along with editors' ratings of the quality of the review.  We also frequently perform literature searches for individuals who have expertise with the research topic under evaluation.

These peer reviewers serve as important consultants.  They provide the editors with information regarding the context and the contribution of the study, in light of existing research, as well as a critical appraisal of the research methods and statistical analyses.

Peer reviewers also provide recommendations about the suitability of the study for publication in the journal, and constructive, objective suggestions to improve the research report.

Comments of the peer reviewers are considered very carefully in editorial decision-making about all research papers, including CAM research.  Physician reviewers are now available for most topics evaluated in CAM research.

In general, decisions to publish CAM studies in a general medical journal, like JAMA, for instance, involve assessment of the quality of the research, the clinical applicability and importance of the research, and the generalizability of the research findings.

An important consideration is the assessment of the merits of the study compared with the information provided in the many other papers submitted for consideration for publication.  As you might expect, in large-circulation, general medical journals, competition for space is extremely keen, particularly keen, with a large number of submitted papers that span the entire gamut of medical research, clinical care, and medical and surgical specialties.

In 2000, the overall acceptance rate for all manuscripts submitted to JAMA was about 10 percent, and for unsolicited manuscripts, the acceptance rate was about 8 percent.  With this competition for space, reports of large-scale, high-quality, multi-center, randomized trials, our highest level of evidence, on clinically important topics, are more likely to receive priority for publication than are single-center studies reporting intermediate or preliminary results about specialized CAM therapies.

In fact, during the past year, of the several studies that we published in JAMA on CAM therapies, most have been randomized trials.

The third question I would like to address, the final, is: What are some of the policy issues regarding publication of CAM research in peer-reviewed journals?

In general, I would support Dr. Campion's policy recommendations, and in addition, would offer several others regarding publication of CAM research in peer-reviewed journals.

Carefully conducted, high-quality, credible research by established biomedical investigators is necessary to evaluate the efficacy and safety of complementary and alternative medicine therapies.  However, until convincing evidence is available that demonstrates safety, efficacy, and effectiveness of these interventions, the uncritical acceptance and widespread application must stop.

With the recent increases in research funding, the establishment of national centers for CAM research, future publication of high-quality studies on CAM therapies should help to establish the evidence base that supports or refutes the safety and efficacy of these many therapies.

Complementary and alternative medicine therapies that are shown to cause, or that are demonstrated to have no beneficial effect should be abandoned.  Decisions regarding the use of and reimbursement for CAM therapies should be based on published evidence and proper cost effectiveness analyses, rather than tradition, anecdotal reports, testimonials, consumer interest, market demand, competition, or political pressures.

Answering fundamental questions about CAM therapies requires critical and objective assessment, using accepted standards for scientific investigation and rigorous evaluation of scientific evidence.  Peer-reviewed medical journals provide one important mechanism for such rigorous evaluation of CAM research, and for providing physicians with reliable information about these diverse and complex therapies.

Dr. Gordon, members of the Commission, thank you very much for the opportunity to present these perspectives on publication of CAM research in peer-reviewed medical journals.  I hope this information is useful in your deliberations on complementary and alternative medicine research policy.  Thank you very much.

DR. GORDON:  Thank you very much.

David Riley.              Presenter: David Riley, M.D.

DR. RILEY:   First, I would like the thank the Commission, Dr. Gordon and the other members of the Commission, for inviting me here.

My name is David Riley.  I am a physician, board-certified in internal medicine and the editor-in-chief of the Journal Alternative Therapies in Health and Medicine.  We began publication in 1995, and were indexed in 1996, and are in the National Library of Medicine.

We view ourselves as a forum for the development of scientific information about the use of alternative therapies and their role in preventing and treating disease, healing illness, and promoting health.  We don't endorse any particular methodology, but we promote the rigorous evaluation of all therapies.

We publish a variety of disciplined inquiry methods, focusing on high-quality scientific research, including randomized, controlled trials, outcome studies, and case reports.

So you might ask the question, why do we need a journal focusing on CAM, and also, is a journal focusing on CAM able to be scientific.  I think a recent letter to "Science" illustrates many of these questions.  It was entitled, "Where Is The Next Einstein?"

It was submitted with a score of signatures from respected academics, including Nobel laureate, Sir Harry Kroto, who is a chemist in England at Sussex, and Dudley Herschbach, a chemist at Harvard, stating that, "All too often the academic research environment favors objectives selected by consensus.  Pioneers and consensus can be very poor bedfellows.  The vogue for expensive, collaborative projects leave little space or money for alternative thinkers.  Thinkers whose research may appear initially irrelevant, but who may come up with stunning work."

I might also add that this letter was rejected to the editors of "Science".  We feel that our journal tries to address this need, and provide a home for some promising and innovative research.

Our journal is by paid subscription only, and the readership consists, primarily, of conventionally licensed health care providers, 60 percent physicians; the balance made up of nurses, other licensed health care providers, naturopaths.  It is interesting that the readership is primarily conventional physicians.

I have some statistics here that I want to go over, briefly, and then I will talk a little bit about the complexity of reviewing articles in a peer-reviewed scientific journal devoted to CAM.

Eighty-seven percent of the original research articles are rejected, 79 percent of brief reports are rejected, 73 percent of hypothesis articles, which is a new section, are rejected -- those statistics could change -- and 64 percent of review articles are rejected.

We have taken an approach that we need a variety of area in the journal to address different types of research.  So we have an Original Article section, which is devoted, primarily, to randomized, controlled trials, although I would say, frankly, that we generally will not get large, randomized, multi-center trials submitted to the journal.  We have, on occasion, and we have one in this issue.  More commonly, we will get smaller studies.

We also have brief reports.  We also have review articles, sometimes solicited.  We also have some other sections, too, that I think are critical.  We have case reports, which are beginning to lay the groundwork for future research that might be done, and I think it is important to have these areas there.

We also have introduced a new section that will start in July on research letters, because, frankly, one of our roles is to educate people who are doing research, and to work with authors so that they can begin to learn how to communicate, because if we are going to communicate with the medical establishment, one of the principles is that you have the communicate in their language, and their language is primarily the peer-reviewed biomedical publication.

So a medical journal that is going to focus on complementary and alternative medicine, by definition, is not designed just for the specialists in a given field.  They have their own forums for communication amongst themselves.  Our goal is to try to communicate the results in complementary and alternative medicine with the biomedical establishment.

I want to talk a little bit about the difficulty of reviewing articles in complementary and alternative medicine, because they can be considerable.  If you look at a conventional medical journal, particularly a specialty journal, there may be a lot of collegiality; everybody knows everybody.  The research areas are fairly well defined.  The unknowns are known.  Frequently, the question becomes one of statistics and methods.

Certainly, in our field, statistics and methods are important.  All articles that are submitted go through a methodological review.  One of our associate editors is a methodologist, and we have several statisticians and methodologists on our editorial board that we turn to regularly, probably too regularly.  That is just the way it is.

We talked a lot about acupuncture research here today.  Let me take that as an example.  If someone submits an article on acupuncture, you need to know, is the author coming from a five-elements school; do they do traditional Chinese medicine; is this a medical acupuncture community; are they doing Korean hand acupuncture.  You name it, there are a of different areas that you need to know.

So not only do you need to find experts in methodology in the given area, but you also have to find an expert in that given therapy.  So you have to pay particular attention.  We have run into issues where, all of a sudden, we realized these two groups that are actually subgroups of a therapy, may not agree with other people about how these things are practiced.

So it can be quite challenging.  Most articles that come in, the original research articles, go through at least revisions, which means there is a review at the end of each revision, because they can be quite complex.  There may be as many as four or five, and I have had even six reviewers on some articles.  Consequently, one of the problems with that is it can delay publication significantly, because each review process, you kind of start anew each time.

So publishing a biomedical journal that is peer-reviewed and indexed, and focusing on complementary medicine is quite challenging, but I also think it is quite necessary, because we need to begin to lay the groundwork for how to communicate with the biomedical establishment.  I also think we need to have a way to begin to publish other kinds of studies.

The conventional medical establishment is much more mature than the complementary and alternative medicine community.  The biomedical model is known.  It is taken for granted.  It is taught in all medical schools.  In complementary and alternative medicine, there are multiple models present.

Just as the randomized, controlled trial is the current best tool we have for investigating a particular type of decision, it is implemented against a background of information, knowledge, and understanding that may not be quite as widespread in complementary and alternative medicine.

I would use the analogy, it is similar to looking at a mural on a wall of building, if you would imagine, with a telescope.  You might get very finely developed information about a very small part of that mural, but you won't understand the mural.

Many of the randomized, controlled trials that are being done, it is difficult to evaluate the results, positive or negative, because you may not actually be looking at an effective evaluation of that treatment.

And so, I think this is a really challenging time and opportunity to begin to develop an evidence profile for complementary and alternative medicine, and to begin to stimulate the next generation of research.  Thank you.

DR. GORDON:  Thank you, David.

Christine Laine.            Presenter: Christine Laine, M.D.

DR. LAINE:   I am Christine Laine and I am the Senior Deputy Editor of Annals of Internal Medicine.  I thank the Commission for the opportunity to present Annals' perspective at this meeting.

A little bit of background about Annals.  Annals is a peer-reviewed biomedical journal that has been published by the American College of Physicians since 1927.  Our circulation is over 100,000, making us the third largest U.S. general medical journal, and the largest medical subspecialty journal, since our focus is on internal medicine.

Our readers are largely internists, but not all.  Some may practice integrative medicine or prescribe CAM therapies themselves, but almost all of them who are clinicians probably take care of patients who seek CAM therapies as well as conventional medical therapies.

Our journal has a high impact factor.  We publish original research reviews, editorials, perspectives, and guidelines.  Aware of the prevalent and increasing use of CAM by people who also seek care from internists, Annals believes strongly that high-quality research and discourse concerning the effectiveness and the safety of commonly used CAM therapies is of great relevance to our readership.

While the number of papers relating to CAM submitted to Annals is increasing, the number remains quite small.  In preparation for this conference, I looked over the information on submitted manuscripts from January 1999 through April of 2001.  Of the, roughly, 6,000 papers submitted to our journal for consideration during that period, only 35 papers related to CAM therapies.

Consequently, we have been developing a series of papers on CAM in close collaboration with two guest editors who are expert in the area.  This series will include somewhere between 15 and 20 papers, and will include a combination of original health services research, critical reviews of the literature, and commentary on a variety of CAM-related issues.

The papers submitted for this series will be subject to Annals usual review process.  Unfortunately, the series will not include original research on specific CAM therapies.  We do anticipate the publication of the series will send out a signal that Annals is interested in the topic and may stimulate investigators in the area to send their original research reports to Annals.

When CAM papers are submitted to Annals, they are subject to our usual rigorous review process.  Unfortunately, many of the papers concerning CAM have important methodological weaknesses, which doesn't make them all that different than many of the papers that concern conventional therapies.

We try to get reviewers that have expertise in alternative medicine, as well as reviewers that have expertise in the particular clinical condition under study.  We have an electronic database of about 10,000 reviewers, but fewer than 100 of the people in that database admit to having expertise in CAM.  So it is a challenge to get reviewers with this expertise.

Let me tell you a little bit about what happened to the 35 papers on CAM submitted to Annals between January '99 and April of 2001.  Two were ultimately published as full articles; one is the letter to the editor; 31 were rejected; and one is in my briefcase to be discussed at our Manuscript Conference tomorrow.

This represents an 8 percent acceptance rate, which is not all that different from Annals' general acceptance rate, which hovers somewhere between 10 and 14 percent, depending on the year.  The acceptance rate for the papers that have been invited to the series is somewhat higher.  We have a special commitment to working with authors through many more rounds of revisions of those papers.  Seventeen have been submitted; nine are accepted; three have been reinvited after substantial revision; three are currently under review; and two have been rejected.

Over recent decades, journals, like the larger medical community, have become increasingly aware of the importance of evidence-based medicine, the growth of large, clinical trials, advances in statistical and analytic approaches, consensus guidelines such as the consolidated standards of reporting trials, and consortia such as the Cochrane Collaboration, are among the many factors contributing to the greater general awareness of the importance of methodological rigor in studies of health care interventions.

In previous times, the medical community gave more weight to anecdotal evidence from case reports, case series, and uncontrolled trials.  In current times, these forms of investigation are considered hypothesis-generating at best.

Unfortunately, much of the CAM-related literature relies on methods that lack the rigor conventional medical journals expect.  This may be in part, and as we have heard today, because CAM practitioners generally do not have the training and education in the experimental process that has become part of the conventional medical education.

Many of the studies of CAM that authors submit to our journal for consideration have important weaknesses such being underpowered, uncontrolled, conducted in non-representative populations, or having inadequate adjustment for potential confounders.

The manuscript that is in my briefcase is one of these.  It is an RCT of a common CAM therapy for a common medical condition.  It is funded by the NIH, HRSA, and industry.  My prognosis is that it is unlikely to make it into publication in Annals, despite having all the trappings that elevate it.  It has federal funding and it is a randomized, controlled trial.

Part of the problems are that they used a preparation that differs substantially from the ones that are in common use in the community.  They studied an atypical study population that doesn't, probably, represent the larger community that uses this therapy; there is a lack of objective outcome measures; and they have marginal statistical power.

Now, the study is reported as negative, but when you look at it closely, it is not truly negative; it is inconclusive.  I think journals take the wrap for being unwilling to publish negative trials.  We have been very aware at Annals, and have one of our statistical editors who has a particular interest in proving that there is no relationship between two things.

So we very carefully look at negative studies.  Most of them are inconclusive, not negative.  Our feeling is, to do a study that is unlikely to give you a conclusive result is, at best, a waste of resources, and, at worst, unethical.

Our suggestions are that we move beyond anecdote and observation to higher levels of evidence; that there is a particular focus on safety; another focus on the interactions between CAM therapies and conventional therapies; and that the CAM therapies be subjected to study with the rigor that the public has come to expect for conventional therapies, realizing that innovative study design and methodologies may be necessary, but that rigor can still exist.  Thank you.

DR. GORDON:  Thank you very much.

Arnold Relman.             Presenter: Arnold Relman, M.D.

DR. RELMAN:   Dr. Gordon and members of the Commission, I cannot speak for the current policies of the New England Journal of Medicine because I retired from the post of editor-in-chief nearly 10 years ago, after serving in that capacity for 14 years.

However, I was familiar with NEJM's policies under its next two editors until the present editor assumed his post, a little less than a year ago.  Given those limitations, I can state confidently that from July 1977, when I first became editor, until July 2000, when the current editor took over, the New England Journal treated manuscripts dealing with CAM in exactly the same manner it treated those concerned with more conventional therapies, and that is the way it should be.

You have just heard Dr. Campion talk about the philosophy of the current editorial leadership, and it appears to be unchanged.  You also heard a very forceful and very articulate statements Drs. Fontanarosa and Laine, speaking for JAMA and Annals, supporting that idea.

Now, I emphasize this because one commonly hears, particularly if you are an editor and you go around to meetings and so on, you commonly hear the complaint that your journal, the New England Journal, or some other leading peer-reviewed journal, is biased against manuscripts dealing with CAM.

Now, all rejected authors tend to have a certain amount of paranoia about their manuscripts and the reasons that they were rejected.  That is part of life.  If you are an experienced editor, as I was for decades, you expect this kind of paranoia and you know how to deal with it, and that is by being absolutely sure you and your staff apply the same standards to all would-be authors and all submissions, including those from your mother, your brother, or from eminent and powerful people in your field who may affect your future in medicine.

If you don't do that, if you apply different standards, you are in deep trouble.  The first advice I have always given to new, young editors who used to come to me and ask me what philosophy do I want to impart to them, is, one, always tell the truth because you can't remember if you lie, and you will be caught sooner or later.  Always tell the truth, and always apply the same standards for the same reason.  You can't live with yourself, and your journal can't survive without that.

So I want to lay to rest the idea that CAM manuscripts, just because they are about alternative medicine modalities, are treated somehow differently.  As far as I know, with the best peer-reviewed journals, and you have heard a few representatives, they are treated the same.

Now, I don't know how many CAM manuscripts we at the New England Journal received during those 23 years that I know something about.  We kept no special file, but there was certainly only a handful of such submissions, and you hear now that there are still only a small number.  In any case, they were handled just like all the other submissions and judged purely on their scientific merits and their suitability for our kind of journal and our readership.

Now, that means that, yes, each editor and each editorial staff has to decide what its readership will be interested in and what will be useful and will determine the tone and the quality of that particular journal.

But given the mounting interest and the growing discussions about its credibility, an editor would be very ill advised to do anything but give CAM submissions at least as much consideration as the others.  I have to tell you that, and I freely admitted my bias, I was looking for good CAM manuscripts to publish.  We got very few, and very few of them met our publishing standards, but I tried very hard to find the best that I could find.  The fact that the journal only published a few, primarily reflects the paucity of really good submissions.

Now, I do not have any statistical data on the relative success rates of CAM papers versus other papers, but you heard some data from Annals.  I just don't know whether it is true of the New England Journal or not, but certainly it was not any conscious editorial policy.

In any case, I think the issue is a fairly straightforward one.  I don't think you need to be a rocket scientist to come to these conclusions.  If CAM research is to have any credibility, it will have to be submitted to the same range of peer-reviewed journals as the more conventional manuscripts dealing with similar illnesses, and it will have to pass the same kind of peer review.  There cannot and there should not be two standards of science.  You have heard that before, nor should there be two standards of peer review.

Unless CAM clinical investigators want to stay outside the mainstream, they must play by the same scientific rules of evidence and publish in the same journals.

So my advice to you is the following:

(1) Emphasize to your constituents and your stakeholders -- that seems to be a popular word.  I don't know what that means, but you know what it means -- your stakeholders, emphasize to them that the time has passed to make arguments based on unsubstantiated claims and undocumented anecdotes; finished, enough of that, enough of the books that claim miraculous cures with no documentation, enough of all the statements in public meetings about marvelous results from a particular case without documentation.

I do not say that anecdotes are useless, and I do not say that anecdotes are not used in ordinary, conventional medical literature; they have been and they should be, but the anecdotes must be documented.  They must be documented.

(2) Encourage the submission of CAM manuscripts to the most rigorously peer-reviewed journals over a wide spectrum, not just to CAM journals but to all journals.  Journals, after all, are not ultimate arbiters of ultimate truth, neither the New England Journal, nor JAMA, nor any other journal of which I am aware, was given by God insight into what ultimate truth is and has the power and has the credibility to say that what we publish is, for sure, true, and what we reject is, for sure, wrong.  Not so.

Journals do the best they can, as Ed Campion said, they do the best they can.  They try to sift out bad evidence from better evidence, and they try to publish the best evidence on the most interesting and the most relevant and important questions for their readership.  That opens the door for lots of journals, but they have to be rigorous.  They have to deal with evidence, quantitative, where possible, objective evidence and documented data.

Finally, my advice to you is to remember that quality, not number of publications, is the important thing.  You don't understand the quality until you read the whole paper, including the fine print, and the methods, and the statistics, and the analysis, and the discussion.  You don't just read the summary, and you don't read somebody else's opinion about what the journal said.

There is no substitute for evidence, and there is no substitute for saying, well, what, actually, did they do; how did they do it; and, what their data are.  When you do that, you often come to different conclusions than those that are generally bandied about.

For example, and I close with this, you heard earlier this morning, that a meta-analysis of homeopathic medicine -- published, I think it was in Lancet.  I read the paper, but I don't remember, was it Lancet, or maybe BMJ -- a meta-analysis showed, beyond any doubt, that homeopathy was effective.

Now, I can't quite remember what was in the abstract, but I know very well what was in the paper, and if you read that paper, it is very clear the data show, and the authors of the paper recognize that the data show, that the best studies, the most rigorous studies, as judged by the analysts, were inconclusive, and the ones that were statistically significant were, by and large, much less rigorous.  A slightly different conclusion from what you heard a little while ago.  Look at the evidence, look at the data, and be guided by the evidence.  Thank you very much.                    Panel Discussion

DR. GORDON:  Thank you.  Thank you, Dr. Relman.

Thank you all.  It is a pleasure to see all of you working so collaboratively with us in addressing these questions.  I feel there has been so much progress made in terms of the dialogue with this field and the major journals.

There was only one thing I was disturbed about, Dr. Relman.  Forty years ago, when I entered Harvard Medical School, I could have sworn they told me God dictated the contents of the New England Journal every week.

[Laughter.]

DR. RELMAN:  Well, you remember, Dr. Gordon, and I was reminded of this just recently in a trip to Jordan, that Moses was denied entrance to the Promised Land, and the New England Journal has not yet gotten there.

[Laughter.]

DR. GORDON:  Thank you for that clarification.

I had one brief question.  As I was listening to a general consensus about the kinds of research that might be published, it occurred to me that one of the places where there might be more attention to CAM or integrative approaches, particularly in the mainstream journals, because I know it is there in the CAM or integrative medicine journals, is in some of the columns that are opinion columns, or the summaries of the literature, or summaries of the clinical experience.

I wonder if you might respond to that, because there has not been very much that I have read in the mainstream journals.

DR. RELMAN:  Is that addressed to me?

DR. GORDON:  Well, you, Dr. Campion, Dr. Laine, DR. FONTANAROSA.

DR. RELMAN:  Well, most of the general medical journals publish a variety of material.  I think, as Dr. Fontanarosa said, we publish not only original such articles, but also reviews of the field, and commentary, and editorial opinion, and so on.  Yes, of course.

I remind you, The New England Journal published the first study by David Eisenberg on the prevalence of alternative medicine use, a very influential study.  We didn't consider it as an original research article in the usual sense.  I think it was a special article.  That is a category we have for thing that have documentation but don't involve the classical kind of research.

DR. GORDON:  No, I know that.  I wasn't asking in an accusatory way.  I was wondering if that might not be a way to open up publication to some of these other issues and deal with them in a thoughtful, critical way.

Phil.

DR. FONTANAROSA:  I would just like to comment on two of the categories of articles you mentioned, one, reviews, and one, commentaries or opinion pieces.  We require a very systematic approach for reviewing articles, to look at the totality of the evidence.

For example, the executive summary of the NCP Cholesterol Guidelines that are published in today's issue of JAMA is the result of a summary and a review of more than 800 references in a 200-page document vetted by 27 experts over about two years.  I think if someone looked at the evidence behind the CAM therapy with that type of rigor, I think a journal would make a mistake not to give it very serious consideration.

On the other extreme, our opinion pieces, or commentaries, I gave you some data for our acceptance rates overall for JAMA.  I will tell you that the acceptance rates for commentaries are even lower than that.

We get a lot of papers that come in to stimulate ideas or thought, or to provoke attention.  Many of those tend to be, quite frankly, not well supported by references, but floating ideas, and we really have to make a determination if we want to spend space on those that would eliminate space from a research paper.

So I would say that the two extremes are rigorous totality of the evidence for reviews, absolutely.  Commentaries or opinion/thought pieces have their place, but they tend to fair less well.

DR. CAMPION:  I would just add that opinion pieces are certainly an appropriate forum for discussion of the issues.

Whenever there is some hot area of controversy in medicine, we tend to hear about from people and get opinion articles.  I am wondering why we are not getting much about alternative medicine and CAM, and I wonder if it is because CAM is actually a lot less controversial than some would say.  It is there; it is accepted by people that use it.

Perhaps people have some perspective on it.  Physicians are well aware of its popularity and may not be as threatened by it as is sometimes said.  Perhaps it is not as controversial.  This does not mean it is accepted as effective, but as a part of our system and of what the consumer wants.

DR. LAINE:  Part of our aim in inviting a series of papers, the papers in the series are much of the type that you describe.  There is a little bit of reviews that are not as critical as we would like, but most of the papers in that series of papers aim to describe the context with which alternative medicine exists in the medical community and the larger society.  So, economic issues, ethical issues, medical, legal issues, that have some relevance to our readership.

So while we are still waiting for the right for the rigorous scientific evidence about specific therapies, we hope that those sorts of articles help conventional physicians understand the context that this whole discipline exists in.

As a sideline, I think there is a perception, like Dr. Relman said, that the conventional medical journals are biased against alternative medicine, papers just reporting on alternative medical therapies.  I don't think we are.  In fact, one of our hopes is that this will show that we are interested, and maybe will stimulate people to send their research to us.  Then, of course, it has to go through the review process.

But one of the reasons we don't publish much, is we don't get much.  The second reason is, of the stuff we get, a lot of it has methodology problems, but that is no different than a lot of the original research that we get on conventional therapies.

DR. GORDON:  Thank you.  My thought was that there may be some articles that are equivalent to discussions of the hypothalamic pituitary adrenal access, for example, summarizing the data that may be interesting, and certainly would be interesting to readers of mainstream medical journals, as well as, perhaps, opinion pieces.

Dean, and Joe, and Effie, and Wayne.

DR. ORNISH:  Well, again, I want to thank the people for such eloquent testimony.  I really appreciate it.

Several people have said that a distinction should not be made between CAM and other medical modalities, either from the standpoint of funding or publication.  I certainly completely share that ideal and your commitment to doing good science.  It is certainly how I spend most of my time, but we have heard testimony from several people who have testified before our Commission that certain journals are more likely to publish an article showing a negative finding from a CAM modality than a positive one.

I am addressing this question to Dr. Campion.  In the three priorities that you outlined in your testimony a few minutes ago, the top one was to give priority to those few complementary and alternative medicine methods that can cause harm.

Do you also have the same commitment to articles that show safety, efficacy, and effectiveness?

DR. CAMPION:  I think, as Dr. Roman outlined, the basic principle is, we will consider any manuscript on it merits.  If the subject is important, if the intervention is important, which means it is something that is in wide use or has potential benefits, we will consider it.  We don't get many of those.  I am not sure if we have gotten any.

We do publish a fair number of negative studies, over 20 percent of the randomized, controlled trials that we publish are negative studies.  Those are of conventional medicine.  We think that negative studies are more important than people realize, both with respect to CAM and with respect to conventional medicine.

DR. GORDON:  Joe.

DR. FINS:  DR. FONTANAROSA uses the phrase "a reputable journal," and as a conventional practitioner, when I see something in the Annals or JAMA or the New England Journal, I feel much more comfortable in feeling that it is true, at least by the standards that I have lived and practiced by.

Dr. Relman, this question is directed to you, but to any of the editors, I know there is a consortium of medical editors that have a working group that have published standards.

Is there any outreach to the CAM journals, or at these other medical journals?  What are the criteria, and how do we know what is a reputable journal?  Because I think that is really the threshold thing.

Yesterday or the day before, we were talking about reimbursement for benefits if things were in a peer-reviewed journal, but the question is, what is a peer-reviewed journal; what is a reputable journal.

How do we establish what the threshold is, and does that organization, that entity of journal editors, have a role in establishing what the cut-off might be?

DR. RELMAN:  Although I was one of the founders of the Vancouver Group, more years ago than I can quite remember, and as of the time I stepped down 10 years ago, there was no attempt to define what was a good journal or what was a bad journal.  The membership in that group was fairly arbitrary, and it didn't have much to do with quality.  It had to do with other relevant editorial things.

So I would ask my colleagues who are now active members of the Vancouver Group whether there is any attempt to define quality, but I would also say that I would be very dubious about any such attempt.  It has got to be very arbitrary.  At the very least, you have to say it ought to be peer-reviewed if it is going to publish original research.

If it says, no, that is not what we are about; we are going to publish commentary, then you don't need, necessarily, peer review, but if you are going to publish original research, it has to be peer-reviewed.  So that would be my only observation.

DR. LAINE:  I have more recent experience than Dr. Relman's with the Vancouver Group, because it met Thursday and Friday of last week in Philadelphia.  So I spent two days with the Vancouver Group.  That isn't its role.  It is a volunteer group of editors from medical journals who came together to create a document called the Uniform Requirements for Manuscripts Submitted to Biomedical Journals to create some standard guidelines that journals can, or choose not to, buy into.

I think, from speaking with people who have been on the Vancouver Group since it started, it sounds like originally it was more focused on the format of the references, and the nuts and bolts of putting together a manuscript.

In its current inclination, in the revision of that document that is in progress, a lot of concern with ethical issues that relate to publication in biomedical journals, but not to give editors some place to refer to when they are trying to sort out difficult issues related to conflicts of interest on the parts of authors, reviewers, and the like, but not in any way setting criteria for what is a good journal and what is a bad journal.

DR. FINS:  It is not unrelated because the other issue is the firewall issue, and advertisements and supplements in some of the CAM journals.

DR. LAINE:  Right.

DR. FINS:  So there may be a role for the Vancouver Group.  For CAM journals, if they were to accept membership in this consortium, it might lift the standards of the ethical issues, the homogeneity of references, and there may be some cross-over effect.

It is just an issue that might be worth --

DR. LAINE:  Right.  That is an interesting point, because one of the things in this document are recommendations about advertising and where it should sit, and where it shouldn't sit, and what types of products are appropriate according to this volunteer group of editors to be in biomedical journals.  That is an interesting point.

MS. WOOTTON:  If I could just make the point that the blue journal, the Journal of Alternative and Complementary Medicine, does not accept advertisements at all.  This was a conscious policy right at the very beginning at its inception because it is such a tricky area for advertisements.  So we are clean.

DR. RELMAN:  Let me interject.  I think that is just great.  If I had had my way when I first became editor of the New England Journal, I would have said to the owners of the journal, live on our modest profits from subscriptions and classified ads.  We were probably the only journal that could have made a profit from just subscriptions and classified ads, but fortunately, not yours.

MS. WOOTTON:  Absolutely.

DR. RELMAN:  Most journals are now caught in the economic vice of advertising income and conflicts of interest.  This is not irrelevant to your concerns.  Conflicts of interest between those who profit from the sale of alternative therapies and alternative devices, and so on, and those who set policies, set government policies, and constitute advisory groups, are rampant, and you ought to be worried about that.

DR. GORDON:  Dr. Campion, please.

DR. CAMPION:  I would just add in response to your question about journal quality, there is no stamp, and there should be none.  Journals can change over time, but journals have to be judged on the process, on what they publish, and on their place in the scholarly community, how readers, the critical audience responds to them.

One indicator of that is the impact factor, how often do other researchers and scholars quote articles from any particular journal.  That is just one rough indicator.

DR. GORDON:  David.

DR. RILEY:  We send all authors the Uniform Requirements for Manuscript Submissions and the Consort Guidelines in terms of publications.  Now, of course, they don't always follow it, and lots of times, that is probably the primary reason for rejection of original research articles.

DR. GORDON:  Effie.

DR. CHOW:  Thank you very much for your input.  I have got many questions, like Wayne usually has, but I will restrict them to one or two.

DR. GORDON:  If you have too many questions, we won't have much lunch.

DR. CHOW:  Regarding Dr. Campion's remark that it may be more accepted than we believe it is, coming from an era in 1970-something when acupuncturists were jailed, and homeopaths were jailed, some of the leaders were jailed, and Christopher Hill from holistic health was jailed, and we were helping to get them out of jail.

I would say it is more accepted now, but it is acceptance on the public level.  Now it seems like we have the challenge of influencing or educating the top level echelon of scientists and policymakers and so forth.  In your reviews of the papers -- 9,000 reviewers, that is really a lot -- you talk about physicians being the managers of CAM processes and so forth.

How much education do you think is enough to educate the physicians and the health professionals who know, really, basically, nothing about energetics or any of the practices?

That is our educational question, too.  This goes into research.  So I am curious, and I think we would like to be enlightened as to, how much education do you think will make a physician and a nurse, a physical therapist, et cetera, qualified to judge the value of a particular practice.

DR. RELMAN:  May I respond to that?  Education has to follow, not precede, the evidence.  You will not influence what the leaders of American medical education teach and what they believe without presenting, without having available persuasive scientific evidence.

That is why it is so important to support research, to publish good research, to teach those who believe in the value of alternative medicine how to do good research.  That is the only way.  There is no other way that you are going to influence the real leaders of American medicine, and there is no other way that you should want to do it.

DR. GORDON:  Dr. Campion, did you want to respond?

DR. CHOW:  We do believe in research.  I just wanted to say that.  That is a big issue here.

DR. CAMPION:  I believe that education of physicians should mainly focus on trying to be better physicians.  There is plenty for physicians to learn already, and there is this huge, poorly defined area of a million different types of CAM, which everybody knows, there is not a lot of objective data there.

Hence, I don't see need for trying to inject the medical curricula with quasi data, or some pretense of data when there isn't any.  I think physicians need to be aware of what their patients want, where their patients go, what other practices are involved, particularly when they come in direct competition.

DR. GORDON:  Phil.

DR. FONTANAROSA:  I would like to respond briefly, if I may.  Your question about how much education is enough is a very difficult question.  Being involved with teaching two courses to the first-year medical students at Northwestern, I can tell you that time in the curriculum is rare for adding on any additional materials.

In fact, if you add something, something else has to get bumped out.  There are so many hours and so many ways that you can do it.  So I would agree with Dr. Relman, that decisions to add things to the curriculum must be based on existing evidence and a good reason for those being introduced, and bumping out something else.

From the standpoint of how much education for physicians in practice, a lot of that is self-directed, of course, by continuing medical education.  I do think that physicians need to be aware of the various CAM therapies, their limitations, their applications, not that they are going to practice them.

I think it would be just as much a mistake for a physician who is not trained in acupuncture to perform it as someone not trained in orthopedic procedures to set a broken wrist -- there needs to be that expertise -- but to be aware of the current information available on these therapies is important so that when patients ask their physicians these questions, they won't just dismiss the question because of their lack of knowledge, and they will be able, more importantly, to see how the CAM therapies might interplay with the conventional therapies, drug/drug interactions and so on.

DR. GORDON:  Wayne.

DR. JONAS:  Thank you for some very, very wonderful comments by everyone.  That was a good discussion and panel.  I hope I can formulate this.  I will put my question first.  How about that?

Do you think anything needs to be done in this area that, we as a federal body trying to look at federal regulations, need to do?

I don't personally see anything that needs to be in this particular area.  It looks like the process is going on like it ought to go on, both with the biases and the lack of evidence.  I think anyone who reads the historical literature and the sociology of science sees that journal publications is the world in which biases are fought out.

We get data and we have editors and reviewers that are biased, and then there is, hopefully, an open and full discussion, and eventually some of the good stuff forms to the top.  There are multiple examples of this.  Parapsychology is fraught with them, chiropractic has documented this recently, Ernst has documented both sides of this problem in publishing, the marked publication bias in favor of positive studies in the complementary medicine journals, very marked.

The reverse of that, the reviewer bias against some complementary therapies among the conventional reviewers.  So I think this exists, and Davidoff [ph], actually, at a previous conference, said this directly.  The question is, is there anything to do about this, because the bottom line is, there isn't very much good research.

So if the goal for all the journals is to get high-quality, rigorous research -- and let that be the arbitrator of this -- if that doesn't exist, is there anything that we have in relation to this, other than what we have already discussed, which is, let's try to get better and more research infrastructure going, and actually outcomes, so that we have a better selection among all journals to pick from?

I am wondering if any of you have an opinion about that.  Is there anything that should be done about these issues, from the publication point of view, anything that we should be doing?

DR. GORDON:  Please, respond.

MS. WOOTTON:  Can I respond to that one?  I think I would go back to the idea of innovative methodology.  This is what the CAM journals can do best, is to nurture new methodologies that are rigorous but bring in new ideas, new approaches

To go back to my recommendations, I think probably the middle one, that there needs to be more research money for practitioner training schools because they are the ones with the knowledge and the insights that can bring forward these appropriate methodologies.

At the moment, there is a dearth of training there.  I think Richard Hammerschlag addressed this one extremely well.  That would be my recommendation.

DR. RELMAN:  Dr. Jonas, I am on your side.  I preach the same sermon that you are preaching.  We need more good evidence.  In the fullness of time, with good evidence and full publication, the reliable will be sorted out from the unreliable; the true from the false.

I would take exception to my colleague, Jackie Wootton's, suggestion that we practitioners ought to be taught how to do research.  That is a special skill.  It requires a different kind of discipline, and neither in so-called conventional medical research, nor in alternative medicine research, I think, can you expect that the average practitioner is going to play an important role.

They can be part of a team, but the team has to be directed and disciplined by people who are professional investigators and who understand the discipline of clinical research.

MS. WOOTTON:  Could I just come back, quickly, to correct any misapprehension.  I am not saying that practitioners should be taught research methodologies.  I am saying that practitioner training schools should be given encouragement to develop new, innovative methodologies, because I think this is where the innovation will come, not by channeling and focusing down the same route that the conventional biomedical research is going in at the moment, whilst at the same time, sustaining the rigor and the basic principles of scientific methodology.

DR. GORDON:  Thank you.

Other responses to this question?  Dr. Fontanarosa?

DR. FONTANAROSA:  Dr. Jonas knows this all too well, from his previous experiences at OAM.  I think that the two places that there might be a way to improve this is being sure that the grant review process is such that there is prioritization of important, answerable research questions.  So clear prioritization of grant funding.

The other seems, probably, easier said than done.  There are, I can't remember how many, national independent centers for CAM research around the country, each sort of focused on its own area of expertise, and I wonder if some better collaboration among these centers might help to answer questions more efficiently.

DR. GORDON:  I wanted to ask you all, too, to follow up on Wayne's question, whether there might not be a place for the editors of some of the journals to give seminars to help some of the investigators develop more appropriate tools for presenting research and papers, a kind of outreach, perhaps, at some of the conferences or some kind of independent meeting.

Would that make sense?

DR. RILEY:  I think if people focus on the Uniform Requirements for Manuscript Submission and the Consort Guidelines.  The Consort Group meets regularly.  There are other guidelines being proposed now, the Moose Guidelines for Outcomes Research.

The information is out there, but it probably needs to be provided more to CAM investigators, or CAM authors when they are writing up data for submission.

DR. GORDON:  I was thinking of something more interactive, actually, a kind of encouragement and a kind of working with on developing papers of higher quality.

DR. LAINE:  I think developing the paper might be too late a step, because once the data are collected, the data are the data.  If the paper has good data and is addressing an important question, as an editor, you can fix up the presentation.  That is a big part of our job, but you can't fix fundamental study design.

So I think there may be a role for the sort of activity you are talking about, but it has to be earlier in the process.

DR. RELMAN:  Dr. Phillips, in describing the program at Harvard, was telling you that one of the things they try to do is train people interested in alternative medicine to learn how to design good studies to find out whether what they are doing works or not.  That should be part of the education of people who want to make a contribution to the field.

DR. GORDON:  Thank you.

Tieraona.

DR. LOW DOG:  It is quick, because everybody is hungry.

DR. LOW DOG:  Thanks for all of your comments.  My question relates to the database and self-described experts, or what they self-describe their expertise in.

There has been wonderful articles in all of these journals, but some mistakes continue to be repeated, especially in the area of botanicals, which I think physicians are interested in because of potential herb/drug interactions, and because they are pharmacologically active substances.

So we get articles that are published that have the wrong species identified.  So we say Siberian ginseng is actually Panix ginseng, which is a completely different genus, or we have an herb/drug interaction article published that talks about hepatotoxicity of echinecea, but having some mistakes around the biochemistry.

So I think that those are errors we wouldn't have in conventional medicine because we are more familiar with the pharmacology.  Have you thought about getting botanists, or trying to expand the database a little bit more, so that they have specialized areas, just to make sure that those errors don't occur?

DR. LAINE:  Yes.  That is something that we constantly do when we go to our reviewer base.  We don't have any process or the resources to go check into people's credentials.  Once they have done a review for us, each reviewer is graded, and that information is included in our database.  Somebody whose done a lousy review for us is unlikely to be chosen to do another review, but we have to rely on what people tell us their area of interest and expertise is.

We do go out and try to find reviewers in CAM therapies, and in other areas, too, where there seems to be a paucity of expertise among our reviewer database, but mistakes will always get by.  I think it is probably just as common in conventional medicine, although in areas where there are lots of reviewers with that background and expertise, it is probably less likely.

Particularly, in alternative therapy, it sometimes seems like we are hitting on the same reviewers over and over again, and then the potential is there for their biases to creep into the reviews, which is another problem, different, but maybe just as great as inaccurate information.

DR. FONTANAROSA:  I would agree with Dr. Laine for the most part, and I would add one thing, that in addition to appropriate reviewers for papers, and this is true of any paper, if there a new mutational analysis for a genetic predisposition, you try to get someone familiar with that methodology, the same thing.

I would also add that editors have a responsibility in reviewing papers, not solely to rely on the reviewers comments, but to do independent work on his or her own, to pool research articles, to try to verify, independently, names of drugs through various textbooks or other resources to find that out.

Once something is published, it is really out there.  There is no pulling it back easily.  So to try to be sure that these T's are crossed, I's are dotted, the right drug is named -- or, the right herb is named, excuse me -- is very important, and I think it is also the editor's responsibility, in addition to the reviewer's information.

DR. GORDON:  Thank you.  Thank you all very much.  This has been a wonderful panel, and we have really enjoyed the discussion and look forward to continuing to work with you all.

We are going to take a break now for lunch.  We will come back at 1:45 for small group meetings.  Small groups will meet until 2:45, and then at 2:50 we will come back here for the large group.

Return to the WHCCAMP Testimony Page