New Orleans—Assessors were unable to give full votes of confidence to any of the four recently evaluated interventional guidelines created by major North American pain medicine societies.

Of further concern, only half of the sample studied was found to be of high methodological quality, and none of the guidelines surveyed adeptly involved all stakeholders such as patients, providers and payors.

Utilizing the validated Appraisal of Guidelines for Research and Evaluation II (AGREE-II), researchers from the University of Toronto assessed the standards from the now-defunct American Pain Society (APS; Spine 2009;34[10]:1066-1077), as well as the American Society of Interventional Pain Physicians (ASIPP; Pain Physician 2013;16[2 suppl]:S49-S283), International Association for the Study of Pain (IASP; Pain 2013;154[11]:2249-2261), and the Canadian Pain Society (CPS; Pain Res Manag 2012;17[3]:150-158).

 

Three separate appraisers trained on AGREE-II reviewed and rated all four guidelines according to 23 items across the six quality domains: scope and purpose, stakeholder involvement, the rigor of development, clarity of presentation, applicability and editorial independence. The assessors then rated each item on a 7-point scale, added all the item scores, and reported the final score as a percentage of the maximum possible score for each domain.

The research was presented at the American Society of Regional Anesthesia and Pain Medicine 2019 fall pain medicine meeting (abstract 184).

Anuj Bhatia, MD, a clinician and researcher in the Department of Anesthesia and Pain Medicine at the University of Toronto, and his team determined a clinical pain guideline to be of “high quality” if it scored a mean scaled score for the rigor of development above 60% and broke that threshold for any two other domains. The overall score was within 0.33 for three of the four guidelines: The APS scored 5.33±1.15, while ASIPP and IASP both scored 5.66±1.15. The lowest score total was that of his own CPS (4.00±1.10). Dr. Bhatia is the chair of the CPS’s Neuropathic Pain Special interest Group.

 

“Currently there is no agreed-to standard to create guidelines, which leads to a common disagreement,” said Timothy Deer, MD, the president and CEO of the Spine and Nerve Center of the Virginias in Charleston, W.Va., and a member of the Pain Medicine News editorial advisory board. “This is complicated by the splintering of the societies that represent pain management and different views and biases that can complicate the conclusions of each guideline.”

In that way, the guidelines are a reflection of the priorities and opinions of the societies who make them, and therefore subject to the same insights and, unfortunately, blind spots of their views. “What surprised us was that the evidence was looked at differently by different guidelines. The same sort of results was interpreted differently based on who the reviewers were,” Dr. Bhatia said.

“I would say that each has some value in context,” Dr. Deer said.

Lower Scores Than Expected

The APS guidelines scored below 50% for both stakeholder involvement and applicability, but over 90% for the rigor of development. ASIPP scored 100% for Scope and Purpose but 0% for Editorial Purpose, “because the editor of the journal where the guideline was published was actually the first author on that paper,” Dr. Bhatia explained, although he noted these standards were the highest scoring across the board.

 

While his team would happily recommend half the sample with little to no modification, and all but the CPS guidelines after some adjustment, Dr. Bhatia told Pain Medicine News that he was still surprised by the results. “You would have thought a large group of people sitting together doing a guideline, that what it produces should be of high quality but it wasn’t,” Dr. Bhatia said.

Although the guidelines’ scores would indicate that most are sound, “what we found that there were some glaring loopholes in the way guidelines have been done,” he said.

Going forward, more needs to be done to ensure editorial independence, Dr. Bhatia said.

“We want to limit the confusion and have more clarity, for if people keep on developing their own guidelines with their small groups, [it] is not going to help the cause,” he said.

 

He also stressed that future guideline development needs to include all relevant stakeholders. Hopefully, “we will have patients, we will have payors, and we will have policymakers in addition to physicians,” he said.

But in the end, it’s the same old trope, he said. “Garbage in, garbage out. If your studies that you’ve done are not very nice,” meaning they are poorly designed or thought out, “the recommendation that come out of those studies are not going to be nice either,” and future guidelines should strive for more robust evidence and more rigorous study designs upon which to base their conclusions.

Age, Location Could Limit Analysis

However, Dr. Bhatia acknowledges that his group’s assessment has limitations. The researchers are all based in an urban environment in eastern Canada. “We did have writers who were geographically from the same area,” he said, “so it is possible that given the limited lack of scope in terms the background of where the people come from, we may have a fairly similar viewpoint.”

 

But he also pointed out the diverse backgrounds of the people involved with careers spanning three continents.

He also noted that no two reviewers interpreted a particular set of guidelines the same way, and he and his co-authors noted the subjectivity of the AGREE-II tool as a shortcoming from the outset.

But the biggest problem with the guidelines may be their age, Dr. Deer said. “After a few years, the value of any guideline is reduced due to new research and publications.”

The guidelines are older published between 2009 and 2013 although no new updates or editions have been published since. Dr. Deer, who is also a former member of ASIPP’s board of directors, believes this makes the assessment unconstructive. “The need to go back in time and find fault in those guidelines seems nonproductive,” he said. “We would benefit more from a summit of concerned parties to establish new standards for future projects and publications.”

And on that second part, Dr. Bhatia would enthusiastically agree: “We’ve been quite keen at coming up with recommendations for our practitioners and our patients as to what kind of procedures they should have to relieve pain.”

As far as the value in looking back, “We decided before we actually write the guidelines, we should see what’s out there.”

—W. Harry Fortuna

Video by Meaghan Lee Callaghan

The sources reported no relevant financial disclosures.