Accident Analysis and Prevention 2020 (May 20); 142: 105580 ~ FULL TEXT
Carol Cancellierea, Eleanor Boylec, Pierre Côté, Lena W. Holm, Louis-Rachid Salmig, J. David Cassidy
Faculty of Health Sciences,
Ontario Tech University,
Oshawa, Ontario, Canada;
Centre for Disability Prevention and Rehabilitation,
Ontario Tech Universty and Canadian Memorial Chiropractic College,
Oshawa, Ontario, Canada.
Importance: The prognosis of post-traumatic headache is poorly understood.
Objective: To develop 0and validate a prognostic model to predict the presence of post-traumatic headache six months after a traffic collision in adults with incident post-traumatic headache.
Design: Secondary analyses of adults with incident post-traumatic headache injured in traffic collisions between November 1997 and December 1999 in Saskatchewan, Canada (development cohort); and between January 2004 and January 2005 in Sweden (validation cohort).
Setting: The Saskatchewan cohort (development) was population-based (N = 4162). The Swedish cohort (validation) (N = 379) were claimants from two insurance companies covering 20 % of cars driven in Sweden in 2004.
Participants: All adults injured in traffic collisions who completed a baseline questionnaire within 30 days of collision. Excluded were those hospitalized >2 days, lost consciousness >30 min, or reported headache <3/10 on the numerical rating scale. Follow-up rates for both cohorts were approximately 80 %.
Predictors: Baseline sociodemographic, pre-injury, and injury factors.
Outcome: Self-reported headache pain intensity ≥3 (numerical rating scale) six months after injury.
Results: Both cohorts were predominantly female (69.7 % in Saskatchewan, 65.2 % in Sweden), with median ages 35.9 years (Saskatchewan), and 38.0 years (Sweden). Predictors were age, work status, headache pain intensity, symptoms in arms or hands, dizziness or unsteadiness, stiffness in neck, pre-existing headache, and lower recovery expectations. With a positive score (i.e., ≥0.75 probability), the model can rule in the presence of post-traumatic headache at six months (development: specificity = 99.8 %, 95 % CI 99.5 %-99.9 %; sensitivity = 1.6 %, 95 % CI 1.0 %-2.6 %; positive likelihood ratio (LR+) = 8.0, 95 % CI 2.7-24.1; negative likelihood ratio (LR-) = 1.0, 95 % CI 1.0-1.0; validation: specificity = 95.5 %, 95 % CI 91.1 %-97.8 %; sensitivity = 27.2 %, 95 % CI 20.4 %-35.2 %); LR+ = 6.0, 95 % CI 2.8-13.2; LR- = 0.8, 95 % CI 0.7-0.8).
Conclusions and relevance: Clinicians can collect patient information on the eight predictors of our model to identify patients that will report ongoing post-traumatic headache six months after a traffic collision. Future research should focus on selecting patients at high risk of poor outcomes (using our model) for inclusion in intervention studies, and determining effective interventions for these patients.
KEYWORDS: Craniocerebral trauma; Neck injuries; Post-traumatic headache; Prognosis.
From the FULL TEXT Article:
According to the International Classification of Headache
Disorders—ICHD-3, post-traumatic headache (PTH) is defined as a new
headache, or worsening of a pre-existing headache beginning within
seven days after head injury, whiplash, craniotomy, or regaining of
consciousness following trauma [Headache Classification Committee of
the International Headache Society, 2013]. Some evidence suggests that
the onset of PTH may be delayed and begin within 30 days of the injury
[Headache Classification Committee of the International Headache
Society, 2013]. The pathophysiology of PTH is poorly understood, but
its etiology likely includes physical and psychological causes [Ferrari
et al., 2005].
PTH is associated with mild traumatic brain injury (MTBI) and
whiplash-associated disorders (WAD) [Ferrari et al., 2005; Cassidy
et al., 2014a]. The annual incidence of MTBI and WAD is approximately
600 per 100,000 persons [Cassidy et al., 2004, 2000]. Individuals with
MTBI and WAD experience a constellation of symptoms known as postconcussion
symptoms (PCS) [Ferrari et al., 2005; Cassidy et al., 2014b;
American Psychiatric Association, 2000; Marshall et al., 2012]. PTH is
the most common PCS and it may persist for up to one year postinjury
in 30 % of individuals [Ferrari et al., 2005; Cassidy et al., 2014b].
Persistent PTH results in personal suffering, disability and impaired
quality of life [Leonardi et al., 2005; Russo et al., 2014]. The World
Health Organization  recognizes headache disorders as a highpriority
public health problem, deserving greater attention in research
and healthcare [Leonardi et al., 2005].
The proportion of individuals affected by PTH after motor vehicle
collisions (MVC) varies between 48 % and 86 %, given heterogeneous
patient populations. [Ferrari et al., 2005; Cassidy et al., 2014a; Lucas
et al., 2014; Sawyer et al., 2015; Crutebo et al., 2010; Partheni et al.,
2000] Whiplash-associated disorders and head injuries are common in
MVCs [Schelp and Ekman, 1990]. However, very little is known about
the prognostic factors of PTH. Therefore, our objective was to develop
and validate a prediction model for reporting PTH six months following
MVC in adults who reported PTH during their baseline interview within
30 days of their collision.
This study complied with the transparent reporting of a multivariable
prediction model for individual prognosis or diagnosis
(TRIPOD) statement [Moons et al., 2015]. It was approved by the Research
Ethics Boards (REBs) of the University of Toronto, and the
University Health Network. The original studies were previously approved
by the REBs of the University of Saskatchewan, and the University
of Alberta (development cohort), and the Regional Committee
on Ethics at Karolinska Institute, Stockholm (validation cohort).
Study design and participants
Development cohort (Saskatchewan): We conducted a secondary
analysis of a population-based incidence cohort study. All adults treated
for motor vehicle injuries in the province of Saskatchewan, Canada, a
province with approximately 743,210 adults (1998) [Saskatchewan
Government, 2020], were followed for one-year after injury (i.e., between
December 1, 1997 and November 30, 1999). Details on the cohort
formation are published elsewhere [Johansson et al., 2015; Cassidy
et al., 2007]. Isolated neck pain after traffic injury is extremely rare;
rather, pain after traffic injury is most commonly reported in multiple
body areas [Ferrari et al., 2005; Johansson et al., 2015; Cassidy et al.,
2007; Hincapie et al., 2010; Côté et al., 2019]. While WAD is typically
labelled as a neck injury, it is best acknowledged as a syndrome that
extends well beyond the neck region [Ferrari et al., 2005; Hincapie et al.,
Validation cohort (Sweden): We conducted a secondary analysis of an
incidence cohort study of personal injury claimants to two Swedish
traffic insurers. Sweden had approximately 7,200,000 adults in 2004
[Statistics Sweden, 2020]. The insurers, Trygg-Hansa and Aktsam, insured
about 20 % of cars driven in Sweden in 2004 [Holm et al., 2008a].
The inception period was between January 15, 2004, and January 12,
2005, and the participation rate was 76 %. Details on the formation of
the cohort are published elsewhere [Holm et al., 2008a]. As with the
Saskatchewan cohort, the Swedish cohort comprised adults reporting
injury in multiple body areas.
Despite some differences in data collected from both cohorts, we
selected Sweden as the validation cohort because of the similarities
between Saskatchewan and Sweden [L. Holm, personal communication,
March 15, 2015]. Both are Western populations. They experienced the
same climates with four distinct seasons of similar lengths, they had
similar car safety laws and speed limits, universal access to health care,
and they had comparable processes for making insurance claims after a
We included all adults who made a bodily injury claim or were
treated by a registered health professional for a motor-vehicle related
injury, and were at least 18 years old at the time of the collision. We
excluded those who (1) did not occupy a motor vehicle at the time of
the collision, (2) completed their baseline questionnaire>30 days after
their collision, (3) reported being hospitalized>2 days as a result of
the collision, (3) reported lost consciousness>30 min after the collision
(Saskatchewan cohort only), and (4) reported that the “accident
caused” headache with intensity<3/10 on the 11-point numerical
rating scale (NRS) at the time of completing their baseline questionnaire.
The Saskatchewan cohort was asked about “average” headache
intensity over the prior week, while the Swedish cohort was asked
about “current” headache intensity.
In both cohorts, baseline data were collected through a self-report
questionnaire, which included sociodemographic factors (e.g., age,
education), preinjury health status (e.g., pre-existing headache), injuries
and other symptoms experienced since the collision (e.g., arm
pain, dizziness), depressive symptomology, and recovery expectations.
Validated measures were used to collect data wherever possible, such as
the NRS for pain intensity, [Jensen et al., 1986] overall general health
(Zajacova and Dowd, 2011), and the Centre for epidemiological Studies
– Depression Scale (CES-D) (Devins et al., 1988).
Follow-up data was collected by computer-assisted telephone interview
in Saskatchewan and self-reported questionnaire in Sweden.
PTH at six months was defined as self-reported headache as a result
of the accident (≥3 NRS) at six months after the collision. Similar to the
baseline questionnaires, the Saskatchewan cohort was asked about
“average” headache intensity over the prior week, while the Swedish
cohort was asked about “current” headache intensity.
Model development and validation
All analyses were conducted using SAS software, version 9.4 (SAS
Institute Inc.). Model development and validation occurred in five
Step 1: candidate predictors. In prediction modelling, predictors
should be selected based on previous studies and clinical knowledge
rather than solely relying on statistical selection methods. [Steyerberg
and Vergouwe, 2014] It is reasonable to include all clinically relevant
predictors in a prediction model despite non-significant univariable
associations. We selected 36 baseline characteristics as candidate
predictors (Table 1) using the scientific literature, clinical knowledge,
and the International Classification of Functioning, Disability and
Health (ICF) model [Leonardi et al., 2005; Moons et al., 2012;
Royston et al., 2009]. Candidate predictors are correlated to, but not
necessarily causal of the target outcome and typically include
sociodemographic factors, health history, and physical examination
findings [Steyerberg, 2009]. We grouped the candidate predictors
according to the five domains of the ICF model: (1) personal factors,
(2) impairments, (3) activity limitations, (4) participation restrictions,
and (5) environmental factors (barriers or facilitators in the physical,
social and attitudinal environment).
In the personal factors domain, female sex has been shown to be
associated with poorer outcomes with respect to PTH, [Sawyer et al.,
2015] WAD [Holm et al., 2008b; Scholten-Peeters et al., 2003; Walton
et al., 2013], and migraine [Buse et al., 2013]. Younger age has been
shown to be associated with worse PTH outcomes [Sawyer et al., 2015],
whereas studies of WAD [Holm et al., 2008b; Scholten-Peeters et al.,
2003; Bohman et al., 2012] and mTBI demonstrated that older age is
associated with worse outcomes [Cassidy et al., 2014b; Carroll et al.,
2004; Scopaz and Hatzenbuehler, 2013]. Being a student has been
shown to be associated with worse outcomes in the mTBI literature
[Carroll et al., 2004; Mann et al., 2004]. The level of education has been
shown to be predictive of a number of health-related outcomes, with
higher education being associated with improved outcomes [Marshall
et al., 2015]. Lifestyle factors (e.g., smoking and alcohol consumption,
diet, and physical activity) [Lantz et al., 2001], and other personal
characteristics (e.g., self-efficacy, coping, pain catastrophizing, fearavoidance
beliefs, and self-esteem) significantly influence health. Selfefficacy
refers to individuals’ assessments of their effectiveness or
competency to perform a specific behaviour successfully required to
produce certain outcomes [Khaw et al., 2008]. It has a beneficial effect
on health behaviour and health status in adults [Bandura, 2004]. Pain
catastrophizing refers to feelings of helplessness, active rumination and
excessive magnification of cognitions and feelings toward the painful
condition [Grembowski et al., 1993]. Ineffective coping mechanisms,
pain catastrophizing, and fear-avoidance beliefs (i.e., fear of movement)
are associated with physical disability in both acute injury and
chronic pain conditions, including whiplash [Leung, 2012; Penley et al.,
2002; Sullivan et al., 2011; Nederhand et al., 2004]. Finally, self-esteem
is a protective factor in physical and mental health [Andersen et al.,
The ICF model characterizes impairments as problems with function
or structure (e.g., signs, symptoms, and abnormal clinical or laboratory
findings). Various physical and psychological impairments have been
shown to be associated with poor outcomes after injury. Initial pain
intensity, such as headache intensity and neck pain intensity was shown
to be associated with poor outcomes after mTBI Cassidy et al., 2014b
and WAD [Holm et al., 2008b; Walton et al., 2013; Bohman et al., 2012;
Carroll et al., 2009a; Williams et al., 2007]. Pain areas, such as arm
pain, have been shown to be associated with poor outcomes after mTBI
[Cassidy et al., 2014b], and pain other than neck and back pain has
been shown to be predictive for recovery after WAD [Bohman et al.,
2012]. Depression, anxiety and stress have also been shown to affect
recovery after mTBI, [Cassidy et al., 2014b] and WAD [Sterling et al.,
2005]. Impairments such as bone fractures, and vision and hearing
problems may contribute to injury-related stress and thus may help to
predict PTH at six months [Nash and Thebarge, 2006]. A wide range of
symptoms may be reported after traffic collisions. For example, associated
with poor outcomes after WAD are dizziness and fatigue
[Williams et al., 2007]. These symptoms are also associated with poor
outcomes after mTBI [Cassidy et al., 2014b; Scopaz and Hatzenbuehler,
2013; Mann et al., 2004]. Loss of consciousness, post-traumatic amnesia,
and disorientation or confusion, are used in diagnosing mTBI.
They have not been shown to be prognostic of recovery after mTBI;
Cassidy et al., 2014b however, it is not known whether any or all of
these signs help predict recovery of PTH. General health and comorbidities
are potentially important predictors of outcomes in adults
with PTH. Individuals with poor health are susceptible to comorbidities
and symptoms. Having poor health or comorbidities is stressful, and
headache could be a side effect of medications [Kuhn et al., 2010]. Preexisting
health conditions or comorbidities such as pre-existing headache,
musculoskeletal problems and mental or emotional problems
have been shown to predict morbidity after trauma including mTBI
[Cassidy et al., 2014b; Sawyer et al., 2015; Hassani-Mahmooei et al.,
2016]. Some researchers also speculate that repeated head injuries
delay the recovery from post-concussion symptoms [Guskiewicz et al.,
Activity limitations and participation restriction are important to
consider as candidate predictors. Activity limitations may occur in the
home, work, or school. Participation restriction refers to problems with
recreational activities, as well as problems with interpersonal relationships
and quality of life. Initial restrictions in activities of daily
living and recreation predicted prolonged symptoms after WAD
[Sterling et al., 2005].
An individual’s physical, social and attitudinal environment should
be considered when attempting to predict health outcomes [Seeman
and Crimmins, 2001]. For example, loud noise or glaring lights in the
physical environment can trigger migraines [Headache Classification
Committee of the International Headache Society, 2013]. Finally, recovery
expectation, influenced in part by society, has been shown to be
predictive of recovery in many conditions including mTBI [Cassidy
et al., 2014b], WAD [Bohman et al., 2012], and cancer [Watson et al.,
2005].Candidate predictors were summarized using frequencies and
percentages. Age was categorized as 18–23, 24–29, 30–39, 4–049,
and ≥50 years [Cassidy et al., 2014a]. Headache pain intensity was
categorized as mild to moderate pain (3–7), and severe pain (8–10)
[Bohman et al., 2012; Fejer et al., 2005; Zelman et al., 2003; Jensen
et al., 2001]. The other pain areas (neck, midback, low back, and face)
were dichotomized (≥3 indicated pain) to remain consistent with our
definition of headache (≥3 NRS).
Step 2: multivariable analysis. We followed best practices to
develop our prediction model by using a combination of clinical
knowledge and data-based methods. In contrast to causal modelling,
in prediction modelling, a simple, robust model may not fit the data
perfectly, but is preferred to a complex, overly fine-tuned model for the
specific data under study. This is because overspecifying a model may
lead to one that may not generalize to new subjects outside the data
under study (i.e., an overfitted model) [Steyerberg and Vergouwe,
2014]. While candidate predictors are considered in the multivariable
analysis, they do not necessarily end up in the final model. Often, a
parsimonious model is preferred if it performs as well as a full model.
We began by selecting all clinically relevant predictors, then used a
data-driven predictor selection strategy, whereby predictors that do not
contribute usefully in the multivariable model are removed [Moons
et al., 2012]. Backward elimination is a preferred strategy for prediction
models, beginning with all candidate predictors and running a sequence
of tests to remove or keep variables in the model [Moons et al., 2012;
Royston et al., 2009]. This is based on a predefined nominal significance
level for variable removal (e.g., 0.10, 0.05, or 0.01). We developed
the model using complete case analysis and multivariable logistic
regression with backward elimination (p-value 0.01 for removal
given the large sample size of the development cohort) [Steyerberg,
2009]. Age and pain intensity did not have a linear relationship with
the logit, thus were categorized into clinically meaningful and interpretable
cut-points as stated above.
We tested two interaction terms based on previous literature,
[Bohman et al., 2012; Carroll et al., 2009a, b] (1) baseline headache
pain intensity and recovery expectations, and (2) baseline headache
pain intensity and depression. We hypothesized that adults would
report poorer recovery expectations and higher levels of depression if
they reported severe, rather than low to moderate, baseline headache
pain intensity. As with other candidate predictors, we included interaction
terms in the model if they remained in the multivariable logistic
regression after backward elimination ((p-value 0.01 for removing
variables) [Steyerberg, 2009]. If multicollinearity was present (variance
inflation factor (VIF)>2.5), the predictor that was judged to be the
least clinically important was removed [Bohman et al., 2012].
Step 3: model evaluation. We reported standard measures of
discrimination: concordance statistic (c-statistic), sensitivity,
specificity, and positive and negative likelihood ratios; and
calibration: the Hosmer-Lemeshow (H–L) test, and calibration plot
[Steyerberg, 2009; Harrell et al., 1996; Hosmer and Lemeshow, 2004;
Steyerberg et al., 2010; Kraft et al., 2009].
Decision-analytic measures are needed if the model is to be used for
making clinical decisions, such as giving patients prognostic estimates,
and deciding on treatment strategies [Steyerberg et al., 2010]. A
probability threshold can be selected to categorize patients as positive
or negative and to weight false positive and false negative classifications.
The harm to benefit ratio defines the relative weight of falsepositive
decisions to true-positive decisions. For example, the probability
threshold may be low if the harm or unnecessary treatment (a
false-positive decision) is relatively limited. On the other hand, if
treatment or overtreatment is harmful, a higher probability threshold
may be selected before making a treatment decision. The ideal
threshold would be one that minimizes bad consequences for those
classified as false positive and false negative. For our model, we decided
that mitigating false positives is more clinically important than reducing
false negatives for predicting poor outcomes for adults with PTH
because there are currently no known effective treatments [Côté et al.,
2015] and overtreatment may do more harm than good [Cassidy et al.,
2007; Côté et al., 2007]. Thus, we opted to categorize patients as "positive"
at a high probability threshold that maximized the specificity of
the model while keeping the prevalence of positive results high.
Step 4: internal validation. The model was internally validated
using random bootstrap resampling with replacement (200 samples)
[Moons et al., 2015].
Step 5: external validation. We used the beta coefficients from
the revised Saskatchewan model that included the characteristics that
were available in the Swedish dataset. In this dataset, information was
not collected for the characteristics midback pain, face pain, unusual
fatigue or tiredness, and current general health status.
Missing follow-up PTH data
In both cohorts, the baseline characteristics were compared between
those who responded to the headache pain intensity question at six
months and those who did not, using the chi-square statistic.
Sensitivity analyses are important to assess the robustness of findings
or conclusions in a study. We conducted sensitivity analyses to
assess the impact of missing outcome data on the overall conclusions of
our study. Missing outcome data can result in selection bias. If, after
performing sensitivity analyses the findings are consistent with those
from the primary analysis and would lead to similar conclusions, we
can be assured that the missing outcome data had little or no influence
on our conclusions. In this case, the results or conclusions are considered
“robust”. Specifically, we tested if imputing missing outcome
values according to three different scenarios lead to different models
and predictive abilities. The three scenarios were: (1) assumed “no
PTH” for those with a missing outcome value (best case scenario); (2)
assumed “PTH” (worst case scenario); and (3) “carry forward the last
observation” (whether at six weeks or three months). We selected the
extremes - best and worst case scenarios - to test the stability of the
model. The multivariable analysis was repeated for each scenario and
the discrimination statistics of the resulting three models were compared
to those resulting from the complete case analysis.
Participants in the development and validation cohorts
The median time between the date of collision and completing the
baseline questionnaire was 10 days (1st to 3rd quartiles 6–16) in the
Saskatchewan cohort, and 22 (1st to 3rd quartiles 17–27) in the Swedish
cohort. The number of eligible participants was 4162 (Saskatchewan)
and 379 (Sweden) (Fig. 1). The final samples included participants with
complete outcome and predictor information (Saskatchewan: n =
2949, 71 %; Sweden: n = 293, 77 %).
Both cohorts were predominantly female (69.7 % Saskatchewan,
65.2 % Sweden). The median age was 35.9 years (1st to 3rd quartiles
25.6–47.2) in Saskatchewan, and 38.0 years (1st to 3rd quartiles
30.0–47.0) in Sweden (Table 1). A larger proportion of participants
reported severe headache pain intensity at baseline in Saskatchewan (n
= 1078, 36.5 %) compared to Sweden (n = 40, 13.6 %). A larger
proportion of participants reported PTH at six months in Sweden (n =
136, 46.4 %) compared to Saskatchewan (n = 939, 31.8 %).
Model development and validation
During assessment for multicollinearity, all VIF values were less
than 2.5, thus there were no problems with multicollinearity.
Multivariable analysis resulted in a prediction model including: age,
work status, headache pain intensity, midback pain, face pain, symptoms
in arms or hands, dizziness or unsteadiness, unusual fatigue or
tiredness, stiffness in neck, poorer current general health, pre-existing
headache, and lower recovery expectations (Table 2). The two interaction
terms did not make it into the model. We could not validate this
full model in the Swedish cohort because four predictor variables were
not available (midback pain, face pain, unusual fatigue or tiredness,
and poorer current general health). Therefore, we tested the performance
of the parsimonious, Swedish-relevant model (with four fewer
predictor variables) in the development cohort prior to externally validating
We opted to categorize patients as "positive" (i.e., will have PTH at
six months) at a high probability threshold that maximized the specificity
of the model while keeping the prevalence of positive results high.
Therefore, we selected a probability threshold of 0.75 because this
value appeared to best meet these criteria in the Swedish-relevant
model (Table 3; see also eTable 2 – full model). This means that anyone
who scores a probability level of 0.75 or higher using our model would
be categorized as "positive" for the likely presence of PTH at six months
after a traffic collision. At this level, the development (Swedish-relevant)
model specificity was 99.8 % (95 % CI 99.5 %–99.9 %) and LR
+ 8.0 (95 % CI 2.7–24.1) (Table 3). The validation model specificity
was 95.5 % (95 % CI 91.1 %–97.8 %) and positive LR + 6.0 (95 % CI
2.8–13.2) (Table 3, Table 4). The full prediction formula is presented
in Table 2. Inserting the correct regression coefficients of observed
values (β) from Table 2 will yield individual probabilities of the outcome.
Further, multiplying the probability by 100 will translate the
probability into percent risk.
Missing data and sensitivity analyses
Participants with missing outcome information (Saskatchewan: n =
1,067, 25.6 %; Sweden: n = 77, 20.3 %) or missing data on predictors
(Saskatchewan: n = 146, 3.5 %; Sweden: n = 9, 2.4 %) were excluded
from the analysis. In general, a higher proportion of participants with
incomplete outcome data in both cohorts reported depressive symptomology,
anxiety, pre-existing headache, and poorer health (eTable 5,
We imputed outcome values for the missing outcomes according to
three different scenarios. The original 8-predictor model (Table 4) and
the three models developed from sensitivity analyses (eTables 7–10)
consisted of between 8 (Swedish-relevant) and 12 (“carry forward last
observation”) predictors. The four models had the majority of predictors
in common (n = 6) – age, severe PTH, symptoms in arms or
hands, stiffness in neck, moderate or severe pre-existing headache, and
lower recovery expectations. Compared to the Swedish-relevant model,
the predictive abilities of the three other models were slightly lower.
The “best case” model was most similar to the Swedish-relevant model,
consisting of all predictors except dizziness or unsteadiness. The “best
case” model also included the predictors poorer current general health,
face pain and unusual fatigue or tiredness.
To the best of our knowledge, we are the first to develop and externally
validate a prediction model to predict PTH six months after a
collision in adults who were identified as having PTH at their baseline
interview. On their own, discrimination and calibration measures such
as the c-statistic, H–L test and calibration plot, are not clinically useful
[Steyerberg, 2009]. Rather, clinicians need to identify who is likely to
have the target disorder which is best expressed by likelihood ratios.
Thus, we selected a threshold value that best suits clinicians’ and researchers’
needs – to rule in PTH and minimizing false positives given
the lack of effective interventions for PTH, as well as identifying people
at high risk of poor outcomes that should be targeted in trials [Côté
et al., 2015]. The magnitude of the positive likelihood ratios in both
cohorts further adds to the value of using our rule for these purposes.
We grouped our predictors according to the ICF model. [Leonardi
et al., 2005] Most of the predictors fall under the ICF domain ‘impairments’,
with ‘personal’ and ‘environmental’ factors also present. Our
results align with prediction models developed for people recovering
after traffic injuries [Bohman et al., 2012; Hartling et al., 2002; Ritchie
et al., 2013; McLean et al., 2014], with respect to overall predictive
accuracy and predictors. For instance, other models generally included
a combination of sociodemographic (e.g., age), psychosocial (e.g.,
posttraumatic stress symptoms, recovery expectations), pre-injury (e.g.,
prior head pain) and physical (e.g., head, neck or back pain) predictors.
Interestingly, in line with other studies on the prognosis of mTBI,
[Cassidy et al., 2014b] factors specific to head injury – LOC and posttraumatic
amnesia - were not predictive of PTH at six months.
There were some differences in data collection between the development
and validation cohorts. First, the time frames for completing
the baseline questionnaire differed (median 10 days in Saskatchewan
vs. median 22 days in Sweden). As we move away from the baseline
period, we are dealing with prevalent headaches. In other words, the
milder incident cases have since improved. Second, there were differences
regarding some of the predictor and outcome definitions and
measurements. For example, with all pain intensity questions, the
Saskatchewan cohort participants were asked about “average” pain
over the past week, while the Swedish cohort participants were asked
about “current” pain. Current pain and average pain are not the same
and average pain over the past week may be a more accurate measure
of pain than current pain [Bolton, 1999]. Third, the outcome data were
collected through computer-aided telephone interviews in the Saskatchewan
cohort, whereas they were collected via a self-report mailed
questionnaire in the Swedish cohort. Computer-aided telephone interviews
are monitored and quality controlled, and may be more accurate
than paper questionnaires. Given these differences between the cohorts,
we expected our model to have lower predictive ability during external
validation. However, our model performed similarly in both cohorts.
Strengths and limitations
The prediction model was developed using a large population-based
incidence cohort. This resulted in stable model estimates [Steyerberg,
2009] leading to similar predictive ability, specificity and positive
likelihood ratio when tested in an independent population. The large
number of events in the development cohort enabled the assessment of
several candidate predictors that have been shown to be related with
health outcomes. The predictors were assessed using psychometrically
sound measures where available. The model was developed using
subject matter knowledge and data-based methods, to achieve a parsimonious
and clinically interpretable model based on predictors that
clinicians can easily collect. Finally, we conducted a geographic validation
study, which is a stringent form of validation [Steyerberg, 2009].
One limitation of our study is the definition of headache. Without a
gold standard diagnosis of PTH, we selected a 30-day cut-off for PTH
onset based on the literature, [Headache Classification Committee of
the International Headache Society, 2013] and we selected a cut-off for
pain intensity (≥3 NRS) to capture clinically important pain [Krebs
et al., 2007]. Adults were excluded if they were hospitalized for more
than two days after the MVC, or if they lost consciousness (LOC) for
more than 30 min. These criteria indicate a more severe injury and have
been used in previous studies of minor traffic injuries [Cassidy et al.,
2014a; Bohman et al., 2012]. LOC was not measured in the Swedish
cohort. However, only a small proportion of people likely satisfied this
criterion since in Saskatchewan, less than 3% reported LOC greater than
Our cohorts were formed 20 years ago, but we are not aware of any
secular or time-related changes in the occurrence or recovery of PTH.
The datasets were not designed to develop or validate the prediction
model; therefore, not all potentially important predictors were available
for inclusion. These may include psychological factors (e.g.,
coping, [Keefe et al., 1992] fear avoidance beliefs, catastrophizing
[Sullivan et al., 2011], and self-efficacy) [Williamson et al., 2008];
lifestyle factors (e.g., smoking, alcohol/substance use, diet, exercise,
and sleep); symptoms of post-traumatic stress disorder (e.g., avoidance
and hyperarousal) [Ritchie et al., 2013], factors related to compensation
[Cassidy et al., 2000], factors related to the physical or social
environment [Leonardi et al., 2005], or perhaps predictors which have
yet to be discovered.
The participation rate in the Swedish cohort was 76 %, potentially
contributing to selection bias and reduced model performance.
However, those who did not participate were more likely not to have
completed the injury claim – supporting the hypothesis that non-responders
had a transient injury or no injury. [Crutebo et al., 2010]. We
had loss to follow up in the development cohort, and conducted sensitivity
analyses testing the extremes (best and worst case scenarios).
We found that three different sensitivity analyses resulted in three
models that were somewhat different from each other and had lower
predictive abilities. Therefore, our model may have preformed more
poorly; however, it performed similiarly in the validation cohort as it
did in the development cohort.
Developers of prediction models must decide whether it is more
important to achieve a high sensitivity or high specificity; high sensitivity
comes at the expense of high specificity and vice versa. Because
we know very little about the effectiveness of treatments for PTH, it is
more important to “rule in” PTH rather than to rule it out [Côté et al.,
2015]. Thus, we decided a priori that we valued reducing false positives,
although this came at the expense of a significant number of false
negatives. We made this decision after considering the consequences of
classification – true positive (those who are classified as having a high
probability of the target outcome that do develop the outcome), true
negative (those who are classified as having a low probability of the
target outcome that do not develop the outcome), false positive (those
who are classified as having a high probability of the target outcome
that do not develop the outcome), and false negative (those who are
classified as having a low probability of the target outcome that do
develop the outcome) [Kraft et al., 2009]. Offering tests and treatments
to all adults with incident PTH after a traffic collision may expose large
numbers of patients with a low probability of poor outcomes to unnecessary
intervention, which can be costly and potentially harmful
[Cassidy et al., 2007; Côté et al., 2007]. A high number of false negatives
would be worrisome if effective interventions are held back from
patients who could benefit. With the current lack of evidence demonstrating
effective treatments for PTH and other common injuries for
people with PTH, such as neck and low back pain [Watanabe et al.,
2012; Varatharajan et al., 2016], identifying true positives may help to
inform a patient of their prognosis and encourage self-management
strategies (e.g., exercise, managing sleep and stress) [Marshall et al.,
2015]. At best, the range of currently available interventions produce
short-term benefits in the form of symptom relief or increased function,
and many commonly used interventions provide no more benefit than
sham or placebo (e.g., pharmacological therapies, manual therapy of
the spine, and cognitive behavioural therapy) [Côté et al., 2015].
Identifying true negatives will help to reassure patients that their
condition will improve. Identifying false negatives may not be too
problematic at this point if there are no known effective treatments.
Finally, identifying false positives may be the most troublesome misclassification,
since treatments may not only be ineffective and costly,
but may also have harmful side effects [Côté et al., 2007]. Indeed, another
analysis using this same Saskatchewan cohort showed that multidisciplinary
rehabilitation of whiplash actually delayed recovery
[Cassidy et al., 2007].
Clinicians can use our model to collect information on the eight
predictors from patients with PTH after a traffic collision. This information
collectively may help clinicians to identify patients who will
have ongoing PTH at six months. Clinicans can provide self-management
strategies to promote healthful lifestyles and symptom management
to all individuals with incident PTH given that these are low-cost,
non-invasive interventions (e.g., proper sleep, nutrition, exercise,
coping, and stress management).. For patients identified as likely to
have ongoing PTH (i.e., are aged over 29 years, students; reported severe
headache, symptoms in arms or hands, dizziness or unsteadiness,
stiffness in neck, moderate/severe pre-existing headache, and uncertain
or poor recovery expectations at the intial interview), clinicians may
monitor them more closely and deliver self-management strategies
more intensely. Furthermore, given that persons injured in traffic collisions
may have injuries and symptoms beyond the head and neck that
may impact PTH outcomes, [Cassidy et al., 2014a; Bohman et al., 2012;
Ritchie et al., 2013] offering evidence-based care for neck and back
pain, for instance, may help to improve outcomes for patients with PTH
[Stulemeijer et al., 2008; Côté et al., 2016]. Thus clinicians should move
beyond a focussed headache assessment even for those who primarily
report headache after a traffic collision. There may be disadvantages for
patients who report ongoing PTH but were not classified as such. For
example, they may experience feelings of discouragement or distress,
and for those who were not self-manageing as well as they might have,
more direct monitoring and education by clinicians might have been
helpful. This may be mitigated by recommending to patients at their
initial visit to contact their healthcare provider if their symptoms are
not improving after the acute period (e.g., 4–6 weeks).
Our model can assist researchers to rule in the presence of PTH six
months after a traffic collision in adults with incident PTH. Researchers
may use our equation to identify participants with a high probability of
ongoing PTH, and assess candidate inteventions in randomized controlled
trials targeted for these individuals. Candidate interventions
may include multimodal interventions that target psychosocial barriers
to recovery such as recovery expectations, lifestyle and behavioural
factors, pain, as well as factors related to activity and participation
Finally, our prediction model may offer a more clinically useful way
of classifying PTH rather than the ICHD-3 definition, Headache
Classification Committee of the International Headache Society, 2013
for which there is no high-quality supporting evidence of its clinical
utility. Classification might be better served if it was based on prognostic
information, rather than an unknown pathophysiology. Clinicians
should focus on improving outcomes for patients by integrating
prognostic information that might affect patient outcomes [Wong et al.,
Post-traumatic headache is common and often persists beyond the
acute period. Our model is useful in helping clinicians predict PTH six
months after a traffic collision in primary care populations. The main
priority for future research is to assess candidate interventions for
people that are at high risk of PTH at six months. This work also suggests
the potential usefulness of a prognostic approach to classifying
PTH beyond the ICHD-3 Headache Classification Committee of the
International Headache Society, 2013.
CC was funded by a Canadian Institutes of Health Research Doctoral
Award (2013–2015). CC completed the first draft, all analyses, results
and interpretation. EB, PC, LWH, LRS, and JDC contributed to all
analyses, results and interpretation. JDC
Declaration of Competing Interest
The authors declare that there are no conflicts of interest.
American Psychiatric Association, 2000.
Diagnostic and Statistical Manual of Mental Disorders (DSM-IV-TR), 4th ed.
APA, Washington, DC.
Andersen, T.E., Karstoft, K.I., Brink, O., Elklit, A., 2016.
Pain-catastrophizing and fearavoidance beliefs as mediators between post-traumatic stress symptoms
and pain following whiplash injury - a prospective cohort study.
Eur. J. Pain 20 (8), 1241–1252.
Bandura, A., 2004.
Health promotion by social cognitive means.
Health Educ. Behav. 31 (2), 143–164.
Bohman, T., Côté, P., Boyle, E., Cassidy, J.D., Carroll, L.J., Skillgate, E., 2012.
Prognosis of patients with whiplash-associated disorders consulting physiotherapy:
development of a predictive model for recovery.
BMC Musculoskelet. Disord. 13, 264.
Bolton, J.E., 1999.
Accuracy of recall of usual pain intensity in back pain patients.
Pain 83 (3), 533–539.
Buse, D.C., Loder, E.W., Gorman, J.A., et al., 2013.
Sex differences in the prevalence, symptoms, and associated features of migraine, probable migraine
and other severe headache: results of the American Migraine Prevalence and Prevention (AMPP) Study.
Headache 53 (8), 1278–1299.
Carroll, L.J., Cassidy, J.D., Peloso, P.M., et al., 2004.
Prognosis for mild traumatic brain injury: results of the WHO Collaborating Centre Task Force on
mild traumatic brain injury.
J. Rehabil. Med. (43 Suppl), 84–105.
Carroll, L.J., Holm, L.W., Hogg-Johnson, S., et al., 2009a.
Course and Prognostic Factors for Neck Pain in Whiplash-associated Disorders (WAD):
Results of the Bone and Joint Decade 2000-2010 Task Force on
Neck Pain and Its Associated Disorders
Spine (Phila Pa 1976). 2008 (Feb 15); 33 (4 Suppl): S83–92
Carroll, L.J., Holm, L.W., Ferrari, R., Ozegovic, D., Cassidy, J.D., 2009b.
Recovery in whiplash-associated disorders: do you get what you expect?
J. Rheumatol. 36 (5), 1063–1070.
Cassidy, J.D., Carroll, L.J., Côté, P., Lemstra, M., Berglund, A., Nygren, A., 2000.
Effect of eliminating compensation for pain and suffering on the outcome of insurance claims for whiplash injury.
N. Engl. J. Med. 342 (16), 1179–1186.
Cassidy, J.D., Carroll, L.J., Peloso, P.M., et al., 2004.
Incidence, risk factors and prevention of mild traumatic brain injury: results of the WHO
collaborating centre task force on mild traumatic brain injury.
J. Rehabil. Med. (43 Suppl), 28–60.
Cassidy, J.D., Carroll, L.J., Côté, P., Frank, J., 2007.
Does multidisciplinary rehabilitation benefit whiplash recovery?: results of a population-based incidence cohort study.
Spine (Phila Pa 1976) 32 (1), 126–131.
Cassidy, J.D., Boyle, E., Carroll, L.J., 2014a.
Population-based, Inception Cohort Study of the Incidence, Course, and Prognosis of
Mild Traumatic Brain Injury After Motor Vehicle Collisions
Arch Phys Med Rehabil. 2014 (Mar); 95 (3 Suppl): S278–285
Cassidy, J.D., Cancelliere, C., Carroll, L.J., et al., 2014b.
Systematic Review of Self-Reported Prognosis in Adults After Mild Traumatic Brain Injury:
Results of the International Collaboration on Mild Traumatic Brain Injury Prognosis
Arch Phys Med Rehabil. 2014 (Mar); 95 (3 Suppl): S132–151
Côté, P., Boyle, E., Shearer, H.M., et al., 2019.
Is a government-regulated rehabilitation guideline more effective than general practitioner education
or preferred-provider rehabilitation in promoting recovery from acute whiplash-associated disorders?
A pragmatic randomised controlled trial.
BMJ Open 9 (1), e021283.
Côté, P., Hogg-Johnson, S., Cassidy, J.D., Carroll, L., Frank, J.W., Bombardier, C., 2007.
Early aggressive care and delayed recovery from whiplash: isolated finding or
Arthritis Rheum. 57 (5), 861–868.
Côté, P., Shearer, H., Ameis, A., et al., 2015.
Enabling Recovery From Common Traffic Injuries: A Focus on the Injured Person.
Financial Services Commission of Ontario
Accessed April 30, 2017.
Côté, P., Wong, J.J., Sutton, D., et al., 2016.
Management of Neck Pain and Associated Disorders: A Clinical Practice Guideline
from the Ontario Protocol for Traffic Injury Management (OPTIMa) Collaboration
European Spine Journal 2016 (Jul); 25 (7): 2000–2022
Crutebo, S., Nilsson, C., Skillgate, E., Holm, L.W., 2010.
The course of symptoms for whiplash-associated disorders in Sweden: 6-month followup study.
J. Rheumatol. 37 (7), 1527–1533.
Devins, G.M., Orme, C.M., Costello, C.G., et al., 1988.
Measuring depressive symptoms in illness populations: psychometric properties of the Center for
Epidemiologic Studies Depression (CES-D) scale.
Psychological Health. 2, 139–156.
Fejer, R., Jordan, A., Hartvigsen, J., 2005.
Categorising the severity of neck pain:
establishment of cut-points for use in clinical and epidemiological research.
Pain 119 (1–3), 176–182.
Ferrari, R., Russell, A.S., Carroll, L.J., Cassidy, J.D., 2005.
A re-examination of the whiplash associated disorders (WAD) as a systemic illness.
Ann. Rheum. Dis. 64 (9),
Grembowski, D., Patrick, D., Diehr, P., et al., 1993.
Self-efficacy and health behavior among older adults.
J. Health Soc. Behav. 34 (2), 89–104.
Guskiewicz, K.M., McCrea, M., Marshall, S.W., et al., 2003.
Cumulative effects associated with recurrent concussion in collegiate football players:
the NCAA Concussion Study.
JAMA 290 (19), 2549–2555.
Harrell Jr., F.E., Lee, K.L., Mark, D.B., 1996.
Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy,
and measuring and reducing errors.
Stat. Med. 15 (4), 361–387.
Hartling, L., Pickett, W., Brison, R.J., 2002.
Derivation of a clinical decision rule for whiplash associated disorders among individuals
involved in rear-end collisions.
Accid. Anal. Prev. 34 (4), 531–539.
Hassani-Mahmooei, B., Berecki-Gisolf, J., Hahn, Y., McClure, R.J., 2016.
The effect of preexisting health conditions on the cost of recovery from road traffic injury:
insights from data linkage of medicare and compensable injury claims in Victoria, Australia.
BMC Health Serv. Res. 16, 162.
Headache Classification Committee of the International Headache Society, 2013.
The international classification of headache disorders, 3rd edition (beta version).
Cephalalgia 33 (9), 629–808.
Hincapie, C.A., Cassidy, J.D., Côté, P., Carroll, L.J., Guzman, J., 2010.
Whiplash injury is more than neck pain: a population-based study of pain localization after traffic injury.
J. Occup. Environ. Med. 52 (4), 434–440.
Holm, L.W., Carroll, L.J., Cassidy, J.D., Skillgate, E., Ahlbom, A., 2008a.
Expectations for Recovery Important in the Prognosis of Whiplash Injuries
PLoS Med. 2008 (May 13); 5 (5): e105
Holm, L.W., Carroll, L.J., Cassidy, J.D., et al., 2008b.
The Burden and Determinants of Neck Pain in Whiplash-associated Disorders after Traffic Collisions:
Results of the Bone and Joint Decade 2000–2010 Task Force on Neck Pain and Its Associated Disorders
Spine (Phila Pa 1976). 2008 (Feb 15); 33 (4 Suppl): S52-59
Hosmer Jr, D.W., Lemeshow, S., 2004.
Applied Logistic Regression.
John Wiley & Sons, New York.
Jensen, M.P., Karoly, P., Braver, S., 1986.
The measurement of clinical pain intensity: a comparison of six methods.
Pain 27 (1), 117–126.
Jensen, M.P., Smith, D.G., Ehde, D.M., Robinsin, L.R., 2001.
Pain site and the effects of amputation pain:
further clarification of the meaning of mild, moderate, and severe pain.
Pain 91 (3), 317–322.
Johansson, M.S., Boyle, E., Hartvigsen, J., Jensen Stochkendahl, M., Carroll, L., 2015.
A population-based, incidence cohort study of mid-back pain after traffic collisions:
factors associated with global recovery.
Eur. J. Pain 19 (10), 1486–1495.
Keefe, F.J., Salley Jr., A.N., Lefebvre, J.C., 1992.
Coping with pain: conceptual concerns and future directions.
Pain 51 (2), 131–134.
Khaw, K.T., Wareham, N., Bingham, S., Welch, A., Luben, R., Day, N., 2008.
Combined impact of health behaviours and mortality in men and women:
the EPIC-Norfolk prospective population study.
PLoS Med. 5 (1), e12.
Kraft, P., Wacholder, S., Cornelis, M.C., et al., 2009.
Beyond odds ratios–communicating disease risk based on genetic profiles.
Nat. Rev. Genet. 10 (4), 264–269.
Krebs, E.E., Carey, T.S., Weinberger, M., 2007.
Accuracy of the pain numeric rating scale as a screening test in primary care.
J. Gen. Intern. Med. 22 (10), 1453–1458.
Kuhn, M., Campillos, M., Letunic, I., Jensen, L.J., Bork, P., 2010.
A side effect resource to capture phenotypic effects of drugs.
Mol. Syst. Biol. 6, 343.
Lantz, P.M., Lynch, J.W., House, J.S., et al., 2001.
Socioeconomic disparities in health change in a longitudinal study of US adults:
the role of health-risk behaviors.
Soc. Sci. Med. 53 (1), 29–40.
Leonardi, M., Steiner, T.J., Scher, A.T., Lipton, R.B., 2005.
The global burden of migraine: measuring disability in headache disorders with
WHO’s Classification of Functioning, Disability and Health (ICF).
J. Headache Pain 6 (6), 429–440.
Leung, L., 2012.
Pain catastrophizing: an updated review.
Indian J. Psychol. Med. 34 (3), 204–217.
Lucas, S., Hoffman, J.M., Bell, K.R., Dikmen, S., 2014.
A prospective study of prevalence and characterization of headache following
mild traumatic brain injury.
Cephalalgia 34 (2), 93–102.
Mann, M., Hosman, C.M., Schaalma, H.P., de Vries, N.K., 2004.
Self-esteem in a broadspectrum approach for mental health promotion.
Health Educ. Res. 19 (4), 357–372.
Marshall, S., Bayley, M., McCullagh, S., Velikonja, D., Berrigan, L., 2012.
Clinical practice guidelines for mild traumatic brain injury and persistent symptoms.
Can. Fam. Phys. 58 (3), 257–267 e128-240.
Marshall, S., Bayley, M., McCullagh, S., et al., 2015.
Updated clinical practice guidelines for concussion/mild traumatic brain injury and persistent symptoms.
Brain Inj. 29 (6), 688–700.
McLean, S.A., Ulirsch, J.C., Slade, G.D., et al., 2014.
Incidence and predictors of neck and widespread pain after motor vehicle collision among
US litigants and nonlitigants.
Pain 155 (2), 309–321.
Moons, K.G., Kengne, A.P., Woodward, M., et al., 2012.
Risk prediction models: I. Development, internal validation, and assessing the incremental value
of a new (bio) marker.
Heart 98 (9), 683–690.
Moons, K.G., Altman, D.G., Reitsma, J.B., et al., 2015.
Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD):
explanation and elaboration.
Ann. Intern. Med. 162 (1), W1–73.
Nash, J.M., Thebarge, R.W., 2006.
Understanding psychological stress, its biological processes, and impact on primary headache.
Headache 46 (9), 1377–1386.
Nederhand, M.J., Ijzerman, M.J., Hermens, H.J., Turk, D.C., Zilvold, G., 2004.
Predictive value of fear avoidance in developing chronic neck pain disability:
consequences for clinical decision making.
Arch. Phys. Med. Rehabil. 85 (3), 496–501.
Partheni, M., Constantoyannis, C., Ferrari, R., Nikiforidis, G., Voulgaris, S., 2000.
A prospective cohort study of the outcome of acute whiplash injury in Greece.
Clin. Exp. Rheumatol. 18 (1), 67–70.
Penley, J.A., Tomaka, J., Wiebe, J.S., 2002.
The association of coping to physical and psychological health outcomes: a meta-analytic review.
J. Behav. Med. 25 (6),
Ritchie, C., Hendrikz, J., Kenardy, J., Sterling, M., 2013.
Derivation of a clinical prediction rule to identify both chronic moderate/severe disability
and full recovery following whiplash injury.
Pain 154 (10), 2198–2206.
Royston, P., Moons, K.G., Altman, D.G., Vergouwe, Y., 2009.
Prognosis and prognostic research: developing a prognostic model.
BMJ 338, b604.
Russo, A., D’Onofrio, F., Conte, F., Petretta, V., Tedeschi, G., Tessitore, A., 2014.
Posttraumatic headaches: a clinical overview.
Neurol. Sci. 35 (Suppl 1), 153–156.
Saskatchewan Government, 2020.
Accessed June 15, 2018.
Sawyer, K., Bell, K.R., Ehde, D.M., et al., 2015.
Longitudinal study of headache trajectories in the year after mild traumatic brain injury:
relation to posttraumatic stress disorder symptoms.
Arch. Phys. Med. Rehabil. 96 (11), 2000–2006.
Schelp, L., Ekman, R., 1990.
Road traffic accidents in a Swedish municipality.
Public Health 104 (1), 55–64.
Scholten-Peeters, G.G., Verhagen, A.P., Bekkering, G.E., et al., 2003.
Prognostic factors of whiplash-associated disorders: a systematic review of prospective cohort studies.
Pain 104 (1–2), 303–322.
Scopaz, K.A., Hatzenbuehler, J.R., 2013.
Risk modifiers for concussion and prolonged recovery.
Sports Health 5 (6), 537–541.
Seeman, T.E., Crimmins, E., 2001.
Social environment effects on health and aging: integrating epidemiologic and demographic
approaches and perspectives.
Ann. N. Y. Acad. Sci. 954, 88–117.
Accessed June 15, 2018.
Sterling, M., Jull, G., Vicenzino, B., Kenardy, J., Darnell, R., 2005.
Physical and psychological factors predict outcome following whiplash injury.
Pain 114 (1–2), 141–148.
Steyerberg, E.W., 2009.
Clinical Prediction Models. A Practical Approach to Development, Validation, and Updating.
Springer-Verlag, New York.
Steyerberg, E.W., Vergouwe, Y., 2014.
Towards better clinical prediction models: seven steps for development and an ABCD for validation.
Eur. Heart J. 35 (29), 1925–1931.
Steyerberg, E.W., Vickers, A.J., Cook, N.R., et al., 2010.
Assessing the performance of prediction models: a framework for traditional and novel measures.
Epidemiology 21 (1), 128–138.
Stulemeijer, M., van der Werf, S., Borm, G.F., Vos, P.E., 2008.
Early prediction of favourable recovery 6 months after mild traumatic brain injury.
J. Neurol. Neurosurg. Psychiatry 79 (8), 936–942.
Sullivan, M.J., Adams, H., Martel, M.O., Scott, W., Wideman, T., 2011.
Catastrophizing and perceived injustice: risk factors for the transition to chronicity after whiplash injury.
Spine (Phila Pa 1976) 36 (25 Suppl), S244–249.
Varatharajan, S., Ferguson, B., Chrobak, K., et al., 2016.
Are non-invasive interventions effective for the management of headaches associated with neck pain?
An update of the bone and joint decade task force on neck pain and its associated disorders
by the Ontario Protocol for Traffic Injury Management (OPTIMa) collaboration.
European Spine Journal, 25(7), 1971–1999
Walton, D.M., Macdermid, J.C., Giorgianni, A.A., Mascarenhas, J.C., West, 2013.
Risk factors for persistent problems following acute whiplash injury:
update of a systematic review and meta-analysis. J. Orthop.
Sports Phys. Ther. 43 (2), 31–43.
Watanabe, T.K., Bell, K.R., Walker, W.C., Schomer, K., 2012.
Systematic review of interventions for post-traumatic headache.
PM R 4 (2), 129–140.
Watson, M., Homewood, J., Haviland, J., Bliss, J.M., 2005.
Influence of psychological response on breast cancer survival: 10-year follow-up of a population-based cohort.
Eur. J. Cancer 41 (12), 1710–1714.
Williams, M., Williamson, E., Gates, S., Lamb, S., Cooke, M., 2007.
A systematic literature review of physical prognostic factors for the development of Late Whiplash Syndrome.
Spine (Phila Pa 1976) 32 (25), E764–780.
Williamson, E., Williams, M., Gates, S., Lamb, S.E., 2008.
A systematic literature review of psychological factors and the development of late whiplash syndrome.
Pain 135 (1–2), 20–30.
Wong, J.J., Côté, P., Sutton, D.A., et al., 2017.
Clinical Practice Guidelines for the Noninvasive Management of Low Back Pain:
A Systematic Review by the Ontario Protocol for Traffic Injury Management
European J Pain 2017 (Feb); 21 (2): 201–216
World Health Organization, 1992.
International Statistical Classification of Diseases and Related Health Problems, 10th ed.
World Health Organization, Geneva, Switzerland.
Zajacova, A., Dowd, J.B., 2011.
Reliability of self-rated health in US adults.
Am. J. Epidemiol. 174 (8), 977–983.
Zelman, D.C., Hoffman, D.L., Seifeldin, R., Dukes, E.M., 2003.
Development of a metric for a day of manageable pain control:
derivation of pain severity cut-points for low back pain and osteoarthritis.
Pain 106 (1–2), 35–42.
Return to HEADACHE