Pages

Monday, September 29, 2014

The AHRQ Draft Report - Fundamentally and irredeemably flawed

The Jury, by John Morgan
The Agency for Healthcare Quality and Research (AHRQ) has released its draft report on the Diagnosis and Treatment of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS).

For those who find it hard to keep track of the plethora of reports, committees, panels, and reviews currently underway, the AHRQ report provides the basis for the Pathways to Prevention, or P2P Workshop. The P2P Workshop has been convened by the NIH for the purpose of making a report that will be used to evaluate research grants for ME/CFS, and upon which pharmaceutical companies will base their clinical trials.

The P2P panel is composed entirely of non-experts. The preference for non-experts was intended, according to Susan Maier, Executive Secretary for the NIH Advisory Committee on Research on Women’s Health, to act like a “jury,” by which she meant the composition of the panel was meant to be unbiased.

Leaving aside for a moment the absurdity of consigning a research case definition for a disease to people who aren't even physicians, the jury system itself does not work. Trial by jury is one of the most inefficient systems of justice in the world. Swayed by bombastic arguments, prejudice, and crocodile tears, juries of twelve unbiased peers routinely find innocent people guilty, and allow the guilty to go free. As a method for determining who gets grants for medical research, anything even remotely resembling the jury system is entirely inappropriate.

I have touched upon some of the short-comings of the draft report below. I've tried to keep it brief. (There is so much wrong with this report that if I were to write a thorough critique, it would be longer than the report itself.)

You can read the AHRQ draft report and appendices here

You can make comments (until October 20) here.

You can read more about the P2P in these posts:

A Review of the P2P Systematic Review
Jennie Spotila gives a great overview of how the AHRQ report will be used, and explains why the draft report is bad science.

All the reasons why the P2P is dangerous

Jennie Spotila explains why the AHRQ report is a recipe for disaster, and how they pulled a “bait and switch” by changing their Key Questions.

Mary Dimmock and Jennie Spotila explain how refusing to consider how CFS and ME differ will affect the results of the P2P Workshop.

Note: Anyone who wishes to reprint this blog post may do so. (Remember to link back to the original and provide attribution.)
_______________________

A BRIEF CRITIQUE OF THE AHRQ DRAFT REPORT

By Erica Verrillo

Point 1: The draft report shows limited understanding of the illness

The report begins with the erroneous statement that “The term ME was first used in the 1930s after an outbreak of neuromyesthenia ...”

A quick google search would have revealed that “myalgic encephalomyelitis” was first used in a letter to the editor entitled, “A New Clinical Entity?” published in the Lancet in 1956. The authors – Emile Nihoul, Lise Quersin-Thiry, S.Chalmers Parry and Robert A. Good – were referring to what became known as Royal Free Disease, an outbreak that occurred in London's Royal Free Hospital in the 1950s. Obviously, none of the members of the panel felt inclined to check their facts.

The report goes on to say that “Uncertainty persists regarding the etiology and whether the condition reflects a single pathologically discrete syndrome, subsets of the same illness, or a nonspecific condition shared by other disease entities.” This statement reflects a thorough misunderstanding of how the illness is perceived by people who investigate and treat it. There are no experts in ME/CFS who would claim that it is a “nonspecific condition shared by other disease entities.” There are, however, numerous researchers who author papers on “chronic fatigue” and “fatiguing illnesses” such as cancer and MS. The authors of the AHRQ draft report, unlike experts in the field, do not have enough background to be able to distinguish between “chronic fatigue” and ME/CFS.

While the introduction to the report is not crucial, it does indicate that the people writing it not only had no knowledge of the illness, but that they did not want to spend any time acquiring it. The willful ignorance that is demonstrated in the introduction permeates the entire report.

Point 2: Problematic Search Methods

In order to find abstracts and articles, the AHRQ searched three main databases using the terms: fatigue; Fatigue Syndrome, Chronic; and Encephalomyelitis. With the notable exception of PsycINFO, a database of abstracts of literature in the field of psychology produced by the American Psychological Association, these are the same databases used by the Drug Class Review: Drugs for Fibromyalgia: Final Original Report published by the Oregon Health & Science University in 2011. Ovid and EBM/Cochrane are large medical databases, though they don't necessarily include every study conducted on a given illness or condition. Only controlled trials are included in the Cochrane databases.

The most glaring problem with the search is that it included studies on “fatigue.” Indeed, a number of studies included in the review were on “fatiguing illnesses” rather than ME/CFS. Like the introduction, the search reflects a state of confusion on the part of the authors. The confusion is not altogether surprising, given that researchers also appear to be confused about the difference between CFS and chronic fatigue. Nonetheless, experts in the field are not confused. They are aware that while ME has been used abroad since the 1950s, it has not been used as a diagnosis here in U.S. Specialists have been limited to CFS as a diagnosis, like it or not. 

A second problem is that with the perennial lack of NIH funding for ME/CFS controlled trials, much of the information about treating the disease is based on clinical observations. None of these were included. nor were studies that were controlled, but which did not meet the set of criteria for inclusion in the review – such as addressing the Key Questions.

Point 3: Studies used for the report are inadequate to address the Key Questions

The studies that the reviewers included were not only too few, they were completely inadequate to properly address the Key Questions.

The Key Questions to be addressed by the report are as follows:

1. What methods are available to clinicians to diagnose ME/CFS and how do the use of these methods vary by patient subgroups?

a) What are widely accepted diagnostic methods and what conditions are required to be ruled out or excluded before assigning a diagnosis of ME/CFS?

b) What is the accuracy and concordance of diagnostic methods?

c) What harms are associated with diagnosing ME/CFS?

2. What are the (a) benefits and (b) harms of therapeutic interventions for patients with ME/CFS and how do they vary by patient subgroups?

a) What are the characteristics of responders and non-responders to interventions?

There are problems with the wording of some of these questions. For example, in a country in which 80% of the physicians don't believe that CFS is a real disease, what could "widely accepted" be referring to? And, "What harms are associated with diagnosing ME/CFS?" seems to have an a priori assumption that diagnosing the disease may in itself cause harm. But aside from the oddness of the wording, the studies they chose do not adequately address the questions.

The criteria for exclusion from the review included, among others, that the study did not last not long enough (therapeutic trial of less than 12 weeks), was published before 1988, had wrong study design, or did not address a Key Question. (There were 8 more exclusions.)

From among the thousands of studies that have been conducted, the criteria limited the review to a scant 64 studies. Some of the landmark studies that were excluded were all of the studies demonstrating immune dysfunction (e.g. NK cell deficiency studies by Brenu et al.), studies of viral reactivation and antiviral treatments (e.g. all Lerner and Jessop studies, Kerr parvovirus B19 study), studies documenting brain abnormalities (e.g. Lange's MRI study), and all of the papers published by Tom Kindlon on harms associated with GET and CBT. Not even appearing on the excluded list were the ground-breaking 2-day CPET studies conducted by Keller, Stevens and Snell, Peckerman's cardiac insufficiency studies, and the recent Watanabe study on CNS inflammation.

The fact that some of the most significant studies in the ME/CFS literature did not even appear on the excluded list was mind-boggling. Of the studies that appeared on the exclusion list, the reasons given were various, but among the most frequently cited were that the studies did not address the Key Questions. Yet, several studies that directly addressed the Key Questions were omitted (for example, 2-Day CPET studies were not even considered), while studies that did not directly address the Key Questions were included. This arbitrariness permeated the entire study selection process.

(Going though the studies that were accepted I found three that did not meet the criteria for inclusion without reading further than the first page. I also found studies in the excluded section that met the criteria. Having no experience with ME/CFS, the panel lacked the ability to distinguish relevant from irrelevant studies.)

Point #4: Contradictory and unsupported conclusions

In terms of treatment, the report was heavily weighted toward psychological studies. Out of the 36 studies used to address Key Question 2, 14 concerned CBT. Considering that the PACE trial was included, but not any of its critiques, it is not surprising that the report favored CBT:
"Based on 13 trials, cognitive and behavioral therapy (CBT), either group or individual; self-instruction booklets; pragmatic rehabilitation: peer-to-peer counseling: and symptom consultation provide improvement in fatigue, function, quality of life, and employment in adult patients with ME/CFS."
Yet, after a detailed examination of the actual results of the trials, the report went on to conclude that:
"In summary, head-to-head trials had mixed results with two trials finding improvement with GET, two trials finding improvement with CBT, and one trial finding no differences between CBT, GET, and usual care. In considering non-head-to-head trial data, there is low strength evidence that CBT and GET provide similar improvement in measures of fatigue and/or functioning."
Those reading this report will base their recommendations on the first statement, as the second statement requires wading through a lot of statistics. They won't even realize that the conclusion that CBT provides "improvement in fatigue, function, quality, and employment" is ultimately derived from the results of a single, deeply manipulated, study. (The PACE trial.)

In terms of Key Question 1, the report suffered from similar inconsistencies, For example, the report concludes with the statement that “the negative effects of being given a diagnosis of ME/CFS appear to be ... universal.”

I could not find a single study from among the list of included studies that would support the conclusion that the diagnosis of ME/CFS had negative effects universally.

In the Asbring and Narvanen study entitled “Women's experiences of stigma in relation to chronic fatigue syndrome and fibromyalgia,” the authors concluded that “ The women experienced stigmatization primarily before receiving a diagnosis.” [Emphasis mine] In addition, they stated that “Stigma consisted of questioning the veracity, morality, and accuracy of patient symptom descriptions and of psychologizing symptoms.”

The Dickson et al. study, “Stigma and the delegitimation experience: An interpretative phenomenological analysis of people living with chronic fatigue syndrome,” reported that “participants reported delay, negotiation and debate over diagnosis: further, they perceived their GPs to be sceptical, disrespectful and to be lacking in knowledge and interpersonal skills.” [Emphasis mine]

In the Green et al. study, “Stigma and Chronic Fatigue Syndrome,” the authors state that “Most subjects (77%) were labeled as ‘psychological cases’ by one or more of the physicians (mean = 8) consulted, but of the 4 stigma measures, only disclosure was related to physician labeling.” This means that patients only felt stigmatized by their physicians after they attempted to educate them about ME/CFS.

There was nothing in these studies to support the claim that the diagnosis itself had negative effects. Rather, it was the delay in diagnosis, and subsequent debate on the part of family and physicians – as well as the delegitimation resulting from a trivializing name – that led to negative effects.

In sum

The conclusions reached in this review were the result of a poorly framed set of key questions, a literature search that managed to exclude the most fundamental research studies, and a misinterpretation of the studies that were eventually deemed acceptable for inclusion. As a whole, this report is fundamentally, and irredeemably, flawed - even given its narrow search results.

These major shortcomings are the inevitable result of appointing a group of people who have no expertise in ME/CFS to evaluate 26 years of research. ME/CFS is a disease that demands expertise. It cannot be evaluated be a panel of non-experts.

To find out what you can do to oppose the P2P, click HERE.