Researching the Appropriateness of Care in the Complementary
and Integrative Health Professions: Part I

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:

FROM:   J Manipulative Physiol Ther. 2018 (Nov); 41 (9): 800–806 ~ FULL TEXT

   Thanks to JMPT for permission to reproduce this Open Access article!   

Ian D. Coulter, PhD, Patricia M. Herman, ND, PhD, Gery W. Ryan, PhD,
Ronald D. Hays, PhD, Lara G. Hilton, PhD, Margaret D. Whitley, BA, MPH

RAND Corporation,
University of California Los Angeles,
Southern California Health Sciences,
Santa Monica, California.

OBJECTIVES:   The purpose of this article is to report on the Center of Excellence for Research on Complementary and Alternative Medicine at RAND Corporation. The overall project examined the appropriateness of chiropractic spinal manipulation and mobilization for chronic low back pain and chronic cervical pain using the RAND and University of California Los Angeles Appropriateness Method, including patient preferences and costs, to acknowledge the importance of patient-centered care in clinical decision-making.

METHODS:   This article is a narrative summary of the overall project and its inter-related components (ie, 4 Research Project Grants and 2 centers), including the Data Collection Core, whose activities and learning will be the subject of a following series of methods articles.

RESULTS:   The project team faced many challenges in accomplishing data collection goals. The processes we developed to overcome barriers may be of use to other researchers and for practitioners who may want to participate in such studies in complementary and integrative health, which previously was known as complementary and alternative medicine.

CONCLUSION:   For this large, complex, successful project, we gathered online survey data, collected charts, and abstracted chart data from thousands of chiropractic patients. The present article delineates the challenges and lessons that were learned during this project so that others may gain from the authors' experience. This information may be of use to future research that collects data from independent practitioners and their patients because it provides what is needed to be successful in such studies and may encourage participation.

KEYWORDS:   Chiropractic; Chronic Pain; Complementary Therapies; Low Back Pain; Manipulation, Spinal; Neck Pain

From the FULL TEXT Article:


Although there is general agreement that all patients should receive health care that is appropriate to their health problem and that inappropriate care is costly, [1] the challenge comes in determining what is appropriate care. [2] In general, appropriateness comprises the right therapy, for the right problem, and for the right patient.

In the current health care system, one answer to the question of appropriateness is that evidence-based care is appropriate care. However, this answer only shifts the problem from deciding what is appropriate to deciding what is evidence-based. Further, there is considerable debate about what percentage of treatments can claim to be evidence based. Some estimate that as little as 15% to 20% of all medical practice can truly claim to be evidence based. [3–6] Hicks notes, “It is generally accepted that between 20% and 60% of patients either receive inappropriate care or are not offered appropriate care.” [6] For large areas of health care, including complementary and integrative health (CIH; previously known as complementary and alternative medicine [CAM]), [7, 8] we have very little data on how much care is appropriate or evidence based.

In the 1980s, the RAND Corporation [2, 8–13] and the University of California, Los Angeles (UCLA), pioneered a method to study the appropriateness of care that not only takes advantage of the available evidence base, but also draws upon the clinical acumen and experience of practitioners. [14] This approach uses a mixed expert and clinician-based panel to consider the available evidence and then judge for a particular treatment whether

“for an average group of patients presenting with this set of clinical indications to an average US physician, the expected health benefit exceeds the expected negative consequences by a sufficiently wide margin that the procedure is worth doing … excluding considerations of monetary cost.” [12]

This has been the most widely used and studied method for defining and identifying appropriate care in the United States, and it also has been used internationally. [15–17] The RAND/UCLA Appropriateness Method (RUAM) makes it feasible to take the best of what is known from research and apply it — using the expertise of experienced clinicians — over the wide range of patients and presentations seen in real-world clinical practice. Clinicians are, after all, the final translators of evidence into practice, and this approach formalizes the process. [15–25]

However, according to the World Health Organization, appropriate care is about ensuring that individuals receive care that is clinically effective; cost-effective; consistent with ethical principles; and meets preferences relevant to individuals, communities, and society. [2, 9] This represents a paradigm shift from previous, narrower definitions of appropriateness that only considered effectiveness, efficacy, and safety. This broader World Health Organization perspective makes explicit that the appropriateness of a procedure can be examined at multiple levels (eg, society, community, individual) and is, in part, dependent on the needs, desires, attitudes, expectations, and preferences of the patients who receive the procedure.

We argue that this broader definition is especially critical for determining the appropriateness of CIH treatments, primarily because CIH users are atypical health care users in several important ways. For example, much of CIH is paid for by the patient out of pocket. It is estimated that CIH utilization amounts to out-of-pocket costs for patients of about $27 billion annually. [10] In addition, most CIH use is consumer-driven, with patients acting as the primary locus of health care integration. [11] But while patients are known to play an important role in driving the expanded use of CIH, little is known about how patients make CIH-related decisions, what their preferences are for types of treatments, or what kinds of results they are seeking and would be satisfied with. If appropriateness of care is ultimately about matching clinically effective and cost-effective treatments with the physical, mental, and emotional needs of affected individuals, then a more thorough understanding of patient-centered desires, expectations, attitudes, and preferences, as well as the cost of these therapies, is required to make health care more effective and more efficient. This study was intended as a step in this direction.

The problem, therefore, for providers, patients, and policymakers is how to decide what is or is not appropriate care. For researchers, it is how to measure appropriateness, how much of health care is appropriate, what effect patient preferences and costs have on appropriateness, and what effect appropriate care has on outcomes.

In 2013, RAND was funded by the National Center for Complementary and Integrative Health to advance the methodology of determining the appropriateness of care in CIH. The target treatments and exemplars for CIH were spinal manipulation and mobilization (M/M), and the target conditions were chronic low back pain (CLBP) and chronic cervical pain (CCP). One important component of this project was the collection of a large amount of data from doctors of chiropractic and their patients.

In this series of articles in the Journal of Manipulative and Physiological Therapeutics, we describe how we gathered the varied and detailed data required to achieve the following study objectives:

(1)   to measure the appropriateness of M/M for CLBP and CCP and

(2)   to determine whether patient preferences and costs affect appropriateness, and further, whether appropriateness affects outcomes.

In each article, we outline the problems we faced with each step of the data collection effort and the methods developed to overcome those problems. By doing so, we provide a blueprint that can be used by others who wish to study the care provided by various types of practitioners in private practice, including those offering CIH. The purpose of this article is to provide an overview of the overall project and its several parts.


This overview of the CERC study attests to the complexity of measuring the appropriateness of complementary and integrative health (CIH) care and its potential modifications even with an established CIH profession such as chiropractic. To steal a political term, it takes a village to do this work. Sixteen research staff were employed on this project, and the total budget for the project was over $8 million.

We learned a lot in this project, and in future papers in the Journal of Manipulative and Physiological Therapeutics we will elaborate on how we were able to bring the data collection portion of this project to fruition. There were many moving parts that needed to be coordinated and integrated, the parts were highly symbiotic, and each element was required to be able to capture the data needed to answer the question of whether and under what circumstances is M/M appropriate for the treatment of chronic low back and neck pain.

The lessons learned here may provide a basis for others who follow, particularly when combined with the detailed information of what we did to achieve our results in following papers in this series. But the lessons are not just for researchers; we hope that they will highlight the extraordinary contribution made by the practice doctors of chiropractic and their staff in this process and encourage future participation. It is only through participation in studies like this that the chiropractic research agenda can be advanced, and only with that can chiropractic fully participate in the world of evidenced-based practice.


This study was done in 6 states of 1 country (United States), thus there is some regional limitation. It also was done in clinics whose practitioners agreed to participate and with patients who agreed to participate. This is acceptable in a center that was funded as a methods center where we are not trying to generalize but are trying to see if this type of research method can be conducted in practices. It also is limited by its focus on chiropractic. Most chiropractic clinics have an organizational structure that includes such things as organized filing systems, including electronic files, computers, and scanning equipment. Although we provided some of this when necessary, for the most part it was possible to work in the clinics with limited disruption. It may be the case that less-established CIH/CAM professions may not have the infrastructures to allow for this. That will need to be discovered in future research.


This article delineates the challenges and lessons that were learned during this project. This information may be of use in future research where data are collected from independent practitioners and their patients because it provides what is needed to be successful in such studies and may encourage participation. There are 3 major conclusions from this report.

  1. Appropriateness studies based on practices can be done in chiropractic and probably other CIH/CAM practices.

  2. Doctors of chiropractic and their staff are not only willing to participate, but also will happily assist in collecting the data and recruiting the patients. They can be trained to participate in quite sophisticated data collection and data protection methods.

  3. Where patients feel the clinic is supportive, they are highly receptive to participating and once enrolled tend to stick with the project.

In following articles of this series, we will provide detailed information on Health Insurance Portability and Accountability Act requirements, survey design, building a practice-based network, provider- and patient-centered research, and chart selection and abstraction.

Practical Applications

  • This paper provides information for CIH researchers who might wish to conduct practice-based research, particularly appropriateness studies.

  • This information also assists those in the chiropractic profession who might want, or be asked, to participate in research.

Funding Sources and Conflicts of Interest

This study was funded by the National Institutes of Health’s National Center for Complementary and Integrative Health Grant No: 1U19AT007912-01. No conflicts of interest were reported for this study.

Contributorship Information

Concept development   (provided idea for the research): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.

Design   (planned the methods to generate the results): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.

Supervision   (provided oversight, responsible for organization and implementation, writing of the manuscript): I.D.C., P.M.H., G.W.R., R.D.H.

Data collection/processing   (responsible for experiments, patient management, organization, or reporting data): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.

Analysis/interpretation   (responsible for statistical analysis, evaluation, and presentation of the results): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.

Literature search   (performed the literature search): I.D.C., M.D.W.

Writing   (responsible for writing a substantive part of the manuscript): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.

Critical review   (revised manuscript for intellectual content, this does not relate to spelling and grammar checking): I.D.C., P.M.H., G.W.R., R.D.H., L.G.H., M.D.W.


  1. Berwick, D and Hackbarth, A.
    Eliminating waste in US health care.
    JAMA. 2012; 307: 1513–1516

  2. Coulter, I, Elfenbaum, P, Jain, S et al.
    SEaRCH™ expert panel process: streamlining the link between evidence and practice.
    BMC Res Notes. 2016; 9: 16

  3. US Congress OoTA.
    Assessing the efficacy and safety of medical technologies. (Available at:); 1978
    Date accessed: January 16, 2019

  4. Michaud, G, McGowan, J, van der Jagt, R et al.
    Are therapeutic decisions supported by evidence from health care research?.
    Arch Intern Med. 1998; 158: 1665–1668

  5. Imrie, R and Ramey, D.
    The evidence for evidence-based medicine.
    Complement Ther Med. 2000; 8: 123–126

  6. Hicks, N.
    Some observations on attempts to measure appropriateness of care.
    Br Med J. 1994; 309: 730–733

  7. Coulter, I.
    Evidence based complementary and alternative medicine: promises and problems.
    Complement Med Res. 2007; 14: 102–108

  8. Coulter, I and Admas, A.
    Consensus methods, clinical guidelines, and the RAND Study of Chiropractic.
    Am Chiro Assoc J Chiropr. 1992; : 52–60

  9. Coulter, I.
    Evidenced-based practice and appropriateness of care studies.
    J Evid Based Dent Pract. 2001; 1: 222–226

  10. Coulter, I.
    Expert panels and evidence: the RAND alternative.
    J Evid Based Dent Pract. 2001; 1: 142–148

  11. Coulter, I, Shekelle, P, Mootz, R et al.
    The use of expert panel results:
    the RAND Panel for Appropriateness of Manipulation and Mobilization of the Cervical Spine.
    Top Clin Chiropr. 1995; 2: 54–62

  12. Shekelle, P.
    The appropriateness method.
    Med Decis Making. 2004; 24: 228–231

  13. Brook, R, Chassin, M, Fink, A et al.
    A method for the detailed assessment of the appropriateness of medical technologies.
    Int J Technol Assess Health Care. 1986; 2: 53–63

  14. Andreasen, P.
    Consensus conferences in different countries.
    Int J Technol Assess Health Care. 1988; 4: 305–308

  15. Stocking, B.
    First consensus development conference in the United Kingdom: on coronary artery bypass grafting.
    Br Med J. 1985; 291: 713–718

  16. Casparie, A, Klazinga, N, van Everdingen, J et al.
    Health-care providers resolve clinical controversies: the Dutch consensus approach.
    Aust Clin Rev. 1987; 7: 43–47

  17. McClellan, M and Brook, R.
    Appropriateness of care: a comparison of global and outcome methods to set standards.
    Med Care. 1992; 30: 565–586

  18. Shekelle, P, Adams, A, Chassin, M et al.
    The Appropriateness of Spinal Manipulation for Low-Back Pain
    Indications and Ratings by a Multidisciplinary Expert Panel

    Santa Monica, CA: RAND Corporation, ; 1992

  19. Fink, A, Brook, R, Kosecoff, J et al.
    Sufficiency of clinical literature on the appropriate uses of six medical and surgical procedures.
    West J Med. 1987; 147: 609–614

  20. Chassin, M. in: Anthony Hopkins (Ed.)
    How Do We Decide Whether an Investigation or Procedure Is Appropriate?
    Royal College of Physicians, London, England; 1989

  21. Leape, L, Park, R, Kahan, J et al.
    Group judgments of appropriateness: the effect of panel composition.
    Qual Assur Health Care. 1992; 4: 151–159

  22. Kahn, KL, Park, RE, Vennes, J et al.
    Assigning appropriateness ratings for diagnostic upper gastrointestinal endoscopy using two different approaches.
    Med Care. 1992; 30: 1016–1028

  23. Shekelle, P, Adams, A, Chassin, M et al.
    The Appropriateness of Spinal Manipulation of Low-Back Pain:
    Indications and Ratings by an All Chirporactic Expert Panel

    in: Report no. 4025/3-CCR/FCER;1992.
    Santa Monica, CA: RAND Corporation; 1992

  24. Shekelle, P, Kahan, J, Bernstein, S et al.
    The reproducibility of a method to identify the overuse and underuse of medical procedures.
    N Engl J Med. 1998; 338: 1888–1895

  25. Coulter, I, Adams, A, and Shekelle, P.
    Impact of varying panel membership on ratings of appropriateness in consensus panels:
    a comparison of a multi- and single-disciplinary panel.
    Health Serv Res. 1995; 30: 577–591

  26. Shekelle, P.G., Hurwitz, E.L., Coulter, I., Adams, A.H., Genovese, B.
    The Appropriateness of Chiropractic Spinal Manipulation for Low Back Pain: A Pilot Study
    J Manipulative Physiol Ther. 1995; 18: 265–270

  27. Shekelle PG, Coulter I, Hurwitz EL, Genovese B, Adams AH.
    Congruence Between Decisions To Initiate Chiropractic Spinal Manipulation
    for Low Back Pain and Appropriateness Criteria in North America

    Annals of Internal Medicine 1998 (Jul 1); 129 (1): 9–17

  28. Coulter, I, Hurwitz, E, Adams, A et al.
    The Appropriateness of Manipulation and Mobilization
    of the Cervical Spine
    Santa Monica, CA: RAND Corporation; 1996 Document No. MR-781-CR.

  29. Coulter, I.
    Manipulation and mobilization of the cervical spine:
    the results of a literature survey and consensus panel.
    J Musculoskelet Pain. 1996; 4: 113–123

  30. Coulter ID, Crawford C, Hurwitz EL, Vernon H, Khorsan R, Suttorp Booth M, Herman PM.
    Manipulation and Mobilization for Treating Chronic Low Back Pain:
    A Systematic Review and Meta-analysis

    Spine J. 2018 (May); 18 (5): 866–879

  31. Herman P, Hilton L, Sorbero ME, et al
    Characteristics of Chiropractic Patients Being Treated for Chronic Low Back and Neck Pain
    J Manipulative Physiol Ther. 2018; 41: 445–455



Since 8-31-2019

                  © 1995–2024 ~ The Chiropractic Resource Organization ~ All Rights Reserved