Brain Injury Newsroom

Specification of Level of Evidence

A three-level system was used to rate individual studies. Well-designed randomized controlled trials (RCTs) were rated as Class I. Studies rated as Class II were RCTs with design flaws; well-done, prospective, quasiexperimental or longitudinal studies; and case-control studies. Case reports, uncontrolled case series, and expert or consensus opinion were generally rated Class III. A well-done, prospective, multicenter or population-based case series can provide valuable information that, in some ways, is more reliable than data from a randomized trial done in a highly selected sample of patients. However, when used to make inferences about effectiveness, an uncontrolled case series is generally classified as Class III, indicating the lowest level of confidence.

A "gray zone" exists between Class II and definite Class III articles. Much of the research in rehabilitation uses quasi-experimental designs. In these observational study designs, control subjects are sometimes identified from a separate patient population. For instance, one group of researchers compared patients undergoing inpatient rehabilitation to a sample of people with TBI who had been treated in a region of the country where formal inpatient TBI rehabilitation was not available. This was an entirely separate patient group, and all the data except outcome measures came from an independent database.

The main difficulty with the quasiexperimental design is lack of control over the constitution of the compared groups. Because there is no randomization and generally no control over the details of the selection process through which the patients received their separate therapies, the groups are likely to differ in the frequency of characteristics that are associated with the outcomes of interest. Even when significant efforts are made to match the experimental and the quasi-control groups, it is likely that significant differences between the groups will remain.

Much of the literature relevant to the five questions addressed in this effort falls into the "gray zone" between Class II and Class III. For this reason, critical appraisal of key studies played a particularly important role in this review. A number of characteristics of these studies were considered relevant to all rehabilitation questions and were recorded in the data abstraction form. Evaluation of the following factors played a major role in critically appraising these articles:

Prospective collection of data.
Complete description of parent study population.
Large study population driven by hypothesis, power, type I error threshold).
Study setting—a single center, many centers, or population based.
Description of reasons for referral to service being studied.
Description of methods sufficiently complete to permit study replication.
Complete description of rehabilitation technique in question (independent variable).
Complete description of differences between "control" and "experimental" groups.
Conditions determining whether patients did or did not receive the rehabilitation technique in question.
Information about potential confounders, including types and severity of injury, age, and others (including in some cases, economic status, educational level, lack of family support).
Measurement of confounding variables using instruments validated as accurate, sensitive, and reliable.
Payer group.
Choice of outcome variables that are meaningful to patients and caregivers.
Use of functional status and other health outcomes rather than surrogate (intermediate) outcomes.
Measurement of outcome variables using instruments validated as accurate, sensitive, and reliable.
Timing of outcome measurements.
Assessment of patient characteristics and outcomes by blinded observer.
Use of multivariate statistical analysis: Were interactions sought and controlled for? Were risk estimates calibrated? Were all relevant confounders included as candidate variables?
The criteria used to classify articles and the features to be considered in critically appraising them were discussed at the subcommittee, committee, national expert panel, and Aspen Neurobehavioral Conference levels with the goal of maintaining consensus at least on the relative stratification of individual articles.

Construction of Evidence Tables
Evidence tables were constructed to summarize the best evidence about effectiveness pertaining to each question. No randomized trials and only a few quasiexperimental studies were available for questions 1 and 2. There were a large number of relevant observational studies of important relationships (for example, the relation of patient characteristics to outcome); studies that concerned individual causal links or relationships in evidence tables were not summarized. For question 3, addressing cognitive rehabilitation, 15 randomized controlled trials and comparative studies that met specified inclusion criteria were placed into evidence tables. All comparative studies located for the last two questions, which addressed supported employment and care coordination, were included in evidence tables.