Trauma Service Guidelines – Appraisal of Evidence and Strength of Recommendations
Introduction
The clinical mission of the Division of Acute Care Surgery is to provide outstanding patient-centered care. 1 To fulfill this mission, the Division aims to create a learning health care system, in “which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the care process… and new knowledge captured as an integral by-product of the delivery experience.” 2
Clinical Practice Guidelines are to be published along with an appraisal of the supporting evidence and a strength of recommendation. Evidence will be critically appraised using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria. Strength of recommendations will be based upon the United States Preventative Services Task Force (USPSTF) method considering level of evidence and net benefit.
Step One – Appraisal of Evidence (GRADE)
Opinion is not evidence. Neither is expert opinion. Guidelines absolutely require expert opinion to help understand the evidence. The guideline, however, must present the evidence underlying those expert opinions in a transparent an unbiased fashion so as to make clear the reasons underlying said opinion.
How do we define “quality of the evidence”? The quality of the evidence refers not just to the internal validity of an individual study (e.g. risk of bias), but to the body of evidence as a whole. The GRADE ratings reflect our confidence that the estimates of effect in the body of evidence as a whole are correct.
This means that a body of evidence comprised of multiple, well-designed clinical trials with minimal risk of bias may be considered low quality due to other factors (e.g. imprecision, inconsistency). This also means that some studies with more than minimal risk of bias (e.g. prospective observational studies) may be upgraded in rating of quality based upon similar factors.
GRADE offers four levels of the quality of the body of evidence: 3
| Level of Quality | Definitions |
|---|---|
| High | We are very confident that the true effect lies close to that of the estimate of the effect |
| Moderate | We are moderately confident that in the estimate of the effect: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different |
| Low | Our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect |
| Very Low | We have very little confidence in the effect estimate: the true effect is likely to substantially different from the estimate of effect |
We the quality of the body of evidence by evaluating the quality of the studies included in the body of evidence collected.
| Study Design – Initial Quality of a Body of Evidence | Lower Quality if: | Higher Quality if: | Final Quality of a Body of Evidence |
|---|---|---|---|
| Randomized Trials – High Quality | Risk of Bias –1 Serious –2 Very serious |
Large effect +1 Serious +2 Very serious |
High |
| Inconsistency –1 Serious –2 Very serious |
Dose response +1 Evidence of a gradient |
Moderate | |
| Observational Studies – Low Quality | Indirectness –1 Serious –2 Very serious |
All plausible residual confounding +1 Would reduce demonstrated effect +2 Would suggest a spurious effect if no effect was observed |
Low |
| Imprecision –1 Serious –2 Very serious |
Very Low | ||
| Publication Bias –1 Serious –2 Very serious |
Step Two – Strength of Recommendation (USPSTF)
After assessing our confidence in the body of evidence, we must rate the strength of the recommendation being made. The USPSTF method assigns a letter grade to recommendations signifying the strength of the recommendation. The letter grade is determined by assessing the
recommendations “Certainty of Net Benefit” and the “Magnitude of Net Benefit.”
“Net benefit” is the balance between benefits and harms and is estimated by evaluating the body of evidence, as described above. Certainty of “net benefit” given one of three levels: 4
| Level of Certainty | Description |
|---|---|
| High | The available body of evidence usually includes consistent results from well-designed, well-conducted studies in representative populations. This conclusion is unlikely to be strongly affected by the results of future studies. |
| Moderate | The available body of evidence is sufficient to determine the effects of the intervention, but confidence in the estimate is constrained by factors such as:
As more evidence becomes available, the magnitude or direction of the observed effect could change and this change may be large enough to alter the conclusion. |
| Low | The available body of evidence is insufficient to assess effects on health outcomes. Evidence is insufficient because of:
More information may allow an estimation of effect on outcomes. |
Once the certainty and magnitude of net benefit has been determined, the USPSTF has a recommendation grid that indicates the strength of recommendation:
| No Data | Magnitude of Net Benefit | |||
|---|---|---|---|---|
| Certainty of Net Benefit | Substantial | Moderate | Small | Zero/Negative |
| High | A | B | C | D |
| Moderate | B | B | C | D |
| Low | I | |||
Strength of Recommendation 5
| A | Intervention is recommended. There is high certainty that the net benefit is substantial. |
| B | Intervention is recommended. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial. |
| C | The intervention is recommended selectively based upon professional judgment and patient preferences. There is at least moderate certainty that the net benefit is small. |
| D | The intervention is not recommended. There is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits. |
| I | Current evidence is insufficient to assess the balance of benefits and harms of the intervention. |
References
1 https://med.uth.edu/about-us/
2 Roundtable on Value & Science-Driven Health Care. The Roundtable. Washington, DC: Institute of Medicine; 2012.
3 Balshem H, et al. GRADE Guidelines: 3. Rating the Quality of Evidence. J Clin Epid. Apr 2011;64(4):401-06.
4 Sawaya GF, et al. Update on the Methods of the USPSTF: Estimating Certainty and Magnitude of Net Benefit. Ann Int Med. Dec 2007(12);147:871-75