When Can We Be Confident about Estimates of Treatment Effects?

Please subscribe to use our print features or to download PDF files.

DR. GLASZIOU: It’s difficult to use the GRADE process, unless you have a systematic review at the starting point, but that makes the work of an individual practitioner or even a guideline committee looking at the evidence, a difficult one because getting up-to-date systematic reviews is a lot of work, and systematic reviews do go out of date reasonably rapidly. One estimate is that approximately 7% of conclusions of the systematic review go out of date each year.

DR. GUYATT: As Dr. Glasziou just emphasized, the GRADE approach starts with an acknowledgment that systematic summaries of the best available evidence are necessary to make a reliable recommendation or a trustworthy recommendation. Selective use of the evidence is potentially very problematic. The practitioners need summaries of the evidence, as has been emphasized: guidelines are one place to obtain them and systematic reviews are another.

What I wanted to emphasize was that systematic reviews can apply the principles that Dr. Montori laid out in terms of the more sophisticated way of judging confidence in estimates, and the GRADE approach that Dr. Schünemann laid out is perfectly applicable to systematic reviews. In fact, ideally, systematic reviews can be summarized looking at each of the issues that the GRADE Working Group has laid out. Indeed, the world’s leading group focused on systematic reviews, whereas the Cochrane Collaboration has adopted the GRADE approach noting the 5 reasons for rating down confidence in estimates and 3 reasons for rating up that Dr. Schünemann described.

Anybody has any final comments or reflections that you’d like to share with our audience?

DR. MONTORI: Individual clinicians and people making recommendations for clinical policy may experience difficulties in understanding this new conceptualization or applying it to their evidence-based ways of practicing. I think it may be pertinent to point out 3 sources that they could use to follow-up this conversation so that they can improve their understanding. Perhaps, we can all discuss the resources they can use for further learning.

DR. GUYATT: Well, Dr. Montori, we’re running short of time. Having introduced this, why don’t you continue?

DR. MONTORI: Well, I would say for those who want to understand the GRADE system that Dr. Schünemann has pointed out, I believe that currently the best place to find that information is on the GRADE Working Group website.3 Clinicians using resources such as UpToDate may find instructions on, for example, how to understand the rating of the condition of evidence that those resources are using. Those are a couple of websites that I would direct clinicians to.

DR. GUYATT: Does anybody else wants to comment on the resources issue?

DR. SCHÜNEMANN: Yes, that is a great point. Additional resources with links on the GRADE Working Group website can be found both on the McMaster GRADE Centre website4 as well as on YouTube, where practitioners might find a series of tutorials that are suited to their needs, starting from brief introductions to very detailed introductions or very detailed descriptions of the GRADE approach.

DR. GLASZIOU: I think it would be great if people could spend time learning how to evaluate and grade evidence themselves. I would probably just warn them that it’s about as difficult as learning how to use a stethoscope, so you have to get a lot of practice and a lot of feedback and guidance to get there. If you don’t feel like you’ve got the time to do that yourself, then you need to find the guidelines or recommendations coming from somebody who has the ability to do it. What I’d advise a clinician to do is to either learn to do it himself/herself or learn to identify resources, guidelines, etc, that have been developed in such a rigorous fashion and have used recommended approaches, such as that of GRADE.

DR. GUYATT: Clinicians need preprocessed information in systematic reviews. Information presented in guidelines is even better for many clinicians because it includes a recommendation and points out where the recommendations are strong and weak. One litmus test for reliability, though not perfect, yet a good indication, is the use of the GRADE approach in summarizing the evidence from a systematic review or in the development of a guideline.