Pages

Friday, February 12, 2016

New NIH Study Raises Questions, Concerns, Hackles

After the NIH posted the protocol for its in-house study on ME/CFS, there was a firestorm of protest, which took Dr. Walter Koroshetz, the man in charge of the new NIH ME/CFS research program, by surprise. 

The protest centered on the use of the Reeves definition (also known as the Empirical Definition) to identify the patient cohort. The Reeves definition has barely been used in research since it was devised in 1995, largely because of its inadequacies. 

There were a number of other problems with the protocol, to which you will find references in the Solve ME/CFS Initiative questions. 

NIH hastily removed the protocol. Then they did something without precedent. Dr. Koroshetz called Carol Head, the CEO of Solve ME/CFS Initiative. 

(Below you will find the questions raised by Solve ME/CFS Initiative after the protocol was posted, as well as Carol Head's summary of her conversation with Dr. Koroshetz.) 

What was clear from the initial protocol was that NIH was on a fishing expedition. Rather than build on previous research, they decided to conduct a vague series of tests (blood tests, stool samples, etc.) in hopes that something would turn up. This is precisely the sort of study we don't need right now. Why reinvent the wheel when there are three decades of research on brain anomalies, immune system dysfunction, mitochondrial dysfunction, and endocrine dysfunction begging for follow-up? (Some of those studies have lain dormant for over a decade for lack of replication.) 

Do we really need to start from scratch - again?

___________________________________

Reprinted with the kind permission of Solve ME/CFS Initiative.

The National Institutes of Health is beginning to recruit participants for its in-house study of ME/CFS patients. The Solve ME/CFS Initiative has identified a number of significant questions and concerns with the design protocol of this research effort. Our organization—represented by our Vice President for Research and Scientific Programs—was immediately in contact with the NIH officials we have an existing, ongoing relationship with to express these serious concerns. We will push forward to determine what may be done to address them and ensure that this study is leveraged to the full benefit of ME/CFS patients. The Solve ME/CFS Initiative also will work in concert with other advocates to ensure maximum impact as a community.

Questions regarding the NIH study protocol that the Solve ME/CFS Initiative will be seeking answers to in the days and weeks to come include:

Is the protocol too broad in its inclusion and as such has little value, or too narrow in that it excludes by design the bulk of relevant patients? Why is it not based on the Canadian Consensus Criteria, which is regarded as the “gold standard” for this complex disease?

Did the protocol examine in-depth the recent advances in the field, including the wealth of information compiled in the 2015 Institute of Medicine report and the slew of commentaries and analysis since, especially on criteria definition beyond the 2003 Reeves criteria? While the study does not rely solely on the Reeves criteria, a clear rationale behind the protocol must be provided. For example, have characteristic ME/CFS symptoms like post-exertional malaise been incorporated under this protocol? If not, why not?

Is this protocol timely and current? In other words, has it benefited from or clearly incorporated the most recent developments in technology, clinical management, basic research or scholarly advances in the field, for instance, the literature and recommendations included in the IOM report?

Has the issue of comorbidity been carefully considered? More specifically, has a clear distinction been made between primary, pathway-specific diagnoses/manifestations versus secondary and pleotropic symptoms like depression or lethargy often associated with a plethora of chronic diseases, such as cancer and diabetes?

Are the study endpoints themselves–both qualitative and quantitative–well defined, established and objective? Additionally, have the number of patient participants been determined according to a bio-statistical analysis for each endpoint? Is the control group the one most relevant to assess the changes in each endpoint or between groups? Are there follow-up plans/alternatives built into the protocol, given its focus on the aspect of immunity and inflammation as an initial stage?

This is the response NIH made to those questions

From Carol Head...

Walter Koroshetz called me moments ago; Zaher Nahle and I had a 20 minute conversation with him and I want to share it.

What was most pointed was his statement that, the NIH often posts protocol-related information online. Most attract zero comment; he is not aware of any posting ever that has attracted the kind of burst of response, all of it negative, that this posting elicited. They were shocked.

While we and others have been telling NIH staff about the intense interest in the ME/CFS community, now it has been clearly illustrated. This gave us the opportunity to explain why: among other things, the intense interest reflects the desperate desire of patients for research progress in this disease. He experienced our decades-long pent-up demand, anger and frustration for federal attention. We also noted that, while virtually all the feedback to NIH was negative, it was HIGHLY informed. We are a patient community that highly attuned to the science and the enormous differences among the several historical diagnostic/clinical criteria.

I told him that some in the patient community plan to boycott the study; we has genuinely mystified. “Why do patients not want ME/CFS research to be done?” We noted that bad research is worse than no research, and “garbage in / garbage out” will occur if the criteria for defining “ME/CFS patients” is not meticulous and highly attuned.

We also stated that funds for this disease MUST come from the federal government; he cited other diseases in which patients have initiated research by amassing significant funds (e.g. he mentioned Huntington’s disease). We discussed the differences in our disease, among them:

  • It is (generally) not fatal (It’s a life sentence, not a death sentence.)
  • There is significant stigma so patients don’t self-identify
  • Patients are often impoverished by this disease
  • It’s difficult to be diagnosed; most are not
  • There are still no clear causes on which to build research budgets.

This makes it quite different from Huntington’s and most other diseases. My sense was that he may not have considered the disease from this “marketing challenge” perspective before, and therefore understand our unique difficulties in raising private research funds; we are so much more dependent on our federal government than most.

He now recognizes that the posting was a significant faux pas; they will post a new protocol. He did not say when.

Dr. Koroshetz has demonstrated his goodwill and genuine desire to move forward in a positive way by proactively calling. He did not have to do so and most probably would not.

Overall, I summarize this as an individual who is committed to doing the right thing and who was shocked by the response. It was/is a wakeup call regarding the intensity of interest and anger in our patient community, and that’s good. We cannot and do not speak for everyone in our patient community (No one can…) and at the same time, I believe that we were able to make incremental progress in closing the enormous gap of understanding that exists between patients and the NIH. I am glad that I was able to contain my longstanding, burning anger long enough to have an intense and candid discussion.

Onward, Carol

Carol Head
President, Solve ME/CFS Initiative
CeHead@SolveCFS.org

Thursday, February 4, 2016

HHS Ignores Request to Review PACE Trial

On February 3, 2016, a group of advocates wrote the Agency for Healthcare Research and Quality(AHRQ) asking them to reconsider the inclusion of the PACE trial in their review.

The AHRQ Evidence Review for ME/CFS formed the basis for the P2P report, in which GET and CBT were reported as beneficial treatments. This conclusion was based on the results of the PACE trial, a study which has been roundly criticized for its flawed methodology.

In November 2015, a group of U.S. organizations sent a letter to the U.S. Health and Human Services (HHS) asking them to address concerns raised in a series of articles about the PACE trial by journalist David Tuller. Based on these concerns and the call by the National Institute of Health (NIH) Pathways to Prevention report to retire the Oxford definition because it could “impair progress and cause harm,” the letter recommended the following steps as appropriate and necessary to protect patients:
  • The AHRQ revise its evidence review to reflect the issues with PACE and with studies using the Oxford case definition in general; 
  • The Centers for Disease Control and Prevention (CDC) remove findings based on PACE and other Oxford case definition studies from current and planned medical education; 
  • HHS use its leadership position to communicate these concerns to other medical education providers; 
  • HHS call for The Lancet to seek an independent reanalysis of PACE.
In the AHRQ’s response, the authors of the evidence review said that they had already considered some of the concerns raised by Tuller and that the additional information would not change the review’s conclusions. This is completely inconsistent with the published review. (The evidence review ranked PACE as a “good” study with “undetected” reporting bias.) AHRQ’s response failed to address the use of the Oxford case definition as the basis of clinical trials for ME/CFS patients. 

The CDC’s response stated that the IOM and P2P “have placed the findings of the PACE trial in an appropriate context for moving the field forward.” (That is bureaucratese for "we are doing nothing.") Like the AHRQ, the CDC failed to address the inclusion of studies based on the Oxford case definition.

HHS did not respond to the request to call on The Lancet to seek an independent review.

If you have not done so, please sign this petition calling for AHRQ and CDC to investigate the PACE trial.

Related Posts



__________________________________________

To: Dr. Arlene Bierman

CC: Dr. Suchitra Iyer, Dr. Wendy Perry

Subject: AHRQ response to community request on PACE and Oxford studies

Date: February 3, 2016

Thank you for your December 24 response to the November 15 patient community letter requesting that AHRQ and CDC investigate the concerns with the PACE trial raised by journalist Dr. David Tuller.1 As you know, this issue is of paramount importance because of the risk of harm to patients from inappropriate treatment recommendations based on flawed studies.

The patient community has requested that AHRQ investigate Dr. Tuller’s concerns and then revise its Evidence Review in light of those concerns and with Oxford definition studies more broadly. The evidence review authors responded that the provided information would not change the conclusions of the report. We disagree.

First, while the authors acknowledge some of the problems with the PACE trial in the full evidence review posted on the AHRQ site, they did not report these problems in the article published in Annals,2 leaving the journal readers unaware of these issues. Of greater concern, in spite of recognizing these issues and stating that they are considered in rating the evidence, the authors still rated PACE as a “good” study with “undetected” reporting bias.3 Such ratings are incompatible with the known flaws in this study and call into question the validity of the evidence based methods used for such a controversial evidence base. Even based on just the information available at the time of AHRQ’s evidence review, the rating of this study and its subsequent impact on the overall treatment conclusions need to be reassessed.

Second, the patient community had also raised concerns with the inclusion of treatment recommendations based on Oxford studies. This problem was highlighted to AHRQ staff when the evidence review protocol was first issued.4 The review itself acknowledged that the Oxford criteria are problematic because Oxford can include patients “with other fatiguing illnesses.” The Pathways to Prevention report stated that the Oxford criteria could “impair progress and cause harm” and called for it to be “retired.”

And yet, the evidence review made general conclusions about the benefits and harms of CBT and GET. For example, the report stated, “GET improved measures of function, fatigue, global improvement as measured by the clinical global impression of change score, and work impairment.” It also concluded that CBT resulted in improvement in physical function scores.

Notably, the evidence review did not qualify these conclusions on treatment effects by case definition. Such statements can reasonably be inferred to apply to all “ME/CFS” patients.5

The obvious question is whether these conclusions would still be true if the Oxford studies had been removed and analyzed separately. In a reply to a published comment on the Annals article raising this issue, the authors acknowledged the importance of analyzing treatment benefits by case definition. They then stated that the improvement in physical function following CBT was seen in Oxford studies, but not in Fukuda studies.6 However, this finding was not stated in the Annals article itself nor did the article report other differences in benefits and harms by case definition. Given that Oxford is acknowledged to include patients with other diseases and given P2P’s call to retire Oxford, the failure to report Oxford findings separate from findings with other case definitions is a serious flaw of this evidence review. The study limitation statements do not compensate for this flaw.

This has real world consequences for patients. The evidence review’s general conclusions about treatment benefits are already being incorporated into clinical guidelines. For instance, referencing AHRQ’s evidence review along with the PACE trial, UpToDate recommends CBT and GET for patients diagnosed by the IOM criteria.7 But CBT and GET have been studied in Oxford cohorts, where they are used to reverse presumed deconditioning, fear of activity, and false beliefs of having an organic disease. Such treatments are obviously inappropriate for the disease that the IOM said is organic, not deconditioning, and characterized by a systemic intolerance to exertion.8 Mixing and matching patient populations in both the evidence review and in clinical guidelines is of questionable medical ethics and creates a significant risk of harm for patients.

We strongly urge AHRQ to work with the evidence review authors to ensure that the PACE trial and its impact on the evidence review’s treatment recommendations is reassessed. Further, it is critical that the authors explicitly report findings for Oxford studies separately from findings in studies using other case definitions. Finally, to ensure that findings in one group of patients are not being harmfully applied to another group of patients, it is essential that the authors explicitly state that treatment conclusions based only on Oxford studies should not be applied to patients meeting other case definitions, particularly those that require post-exertional malaise.

This matter is of the utmost importance to patients. We hope you will give it the full attention that it deserves. Contact Mary Dimmock if you need additional clarification or background.

Signed

Massachusetts CFIDS/ME & FM Association
The Myalgic Encephalomyelitis Action Network (#MEAction)
MEadvocacy.org
New Jersey ME/CFS Association, Inc.
Open Medicine Foundation (OMF)
Phoenix Rising
Solve ME/CFS Initiative
Pandora Org
Wisconsin ME and CFS Association, Inc.
Mary Dimmock
Claudia Goodell
Denise Lopez-Majano (Speak Up About ME )
Matina Nicolson
Donna Pearson
Jennifer Spotila JD (OccupyCFS)
Meghan Shannon
Erica Verrillo (Onward Through the Fog)

1 Patient organizations’ letter to AHRQ and CDC on PACE and Oxford definitions. November 15, 2015. https://dl.dropboxusercontent.com/u/89158245/CDC-AHRQ%20Request%20PACE%20Nov%202015.pdf

AHRQ response to November 15, 2015 letter. December 24, 2015. https://dl.dropboxusercontent.com/u/89158245/AHRQ%20response%20to%20PACE%20request%20-%20Dec%2024%202015.pdf

David Tuller. “TRIAL BY ERROR: The Troubling Case of the PACE Chronic Fatigue Syndrome Study.” Virology Blog. October 21-23, 2015.

First installment: http://www.virology.ws/2015/10/21/trial-by-error-i/

Second installment: http://www.virology.ws/2015/10/22/trial-by-error-ii/

Third installment: http://www.virology.ws/2015/10/23/trial-by-error-iii/

2 Smith MB, Haney E, McDonagh M, Pappas M, Daeges M, Wasson N, et al. “Treatment of Myalgic

Encephalomyelitis/Chronic Fatigue Syndrome: A Systematic Review for a National Institutes of Health Pathways to Prevention Workshop.” Ann Intern Med. 2015; 162: 841-850. http://dx.doi.org/10.7326/M15-0114

The full report is posted on the AHRQ site at this location: http://effectivehealthcare.ahrq.gov/ehc/products/586/2004/chronic-fatigue-report-141209.pdf

3 Ibid.

• Appendix E: Quality Rating Criteria states that in a “good” study, for instance, “important outcomes are considered.” But as Tuller reported, outcomes were changed midtrial, criteria were changed, and some outcomes/analyses were not reported.

• Appendix F: Strength of Evidence Criteria defines reporting bias, which it states includes: ”Study publication bias, i.e., nonreporting of the full study; Selective outcome reporting bias, i.e., nonreporting (or incomplete reporting) of planned outcomes or reporting of unplanned outcomes; and Selective analysis reporting bias, i.e., reporting of one or more favorable analyses for a given outcome while not reporting other, less favorable analyses.”

• Appendix K: Strength of Evidence tables reported Reporting bias as “Undetected” in rows that included PACE, even when the only study covered in a particular outcome was PACE.

4 The authors state that they correctly included the Oxford criteria because the protocol specified it. However, the problems with inclusion of Oxford were raised immediately after the protocol was issued on May 1, 2014.

• Spotila, J. “Protocol for Disaster.” OccupyCFS. May 3, 2014. http://www.occupycfs.com/2014/05/02/protocolfor-disaster/

• M. Dimmock contacted Dr. Beth Collins Sharpe (of AHRQ and ex-officio to CFSAC at the time) by email on May 4, 2014 with this concern. On 6/4/2014, Collins-Sharpe responded that she didn’t “think that the different diagnoses will be lumped together for analysis.” She added, “You’re right that it would be comparing Oxford apples to CCC oranges.“ But then the authors lumped together all definitions in the analyses.

Additionally, this issue was raised directly with Dr. Collins

• Spotila J, Dimmock M. Letter submitted to Dr. Francis Collins, Director of NIH, regarding Pathways to Prevention Workshop on ME/CFS.” May 28, 2014.

https://dl.dropboxusercontent.com/u/89158245/Dr%20Collins%20P2P%20Letter%20Public%20052814%20w%20attachments.pdf

See attachment 2 (Page 20) discussed the Review Protocol and stated. “At least one case definition
[Oxford] requires no more than unexplained fatigue for a diagnosis of CFS, despite the mounting evidence that such case definitions capture a different study population than definitions that require post-exertional malaise, cognitive dysfunction or other multisystem impairments.“

5 UpToDate is one example where the conclusions of this evidence review have been interpreted as applying to all patients.

It is important to note that PACE trial publications have said that the PACE trial findings also apply to patients defined by the CDC CFS criteria and the London ME criteria. But this patient characterization was done after first selecting patients by Oxford. As Dr. Bruce Levin said in Dr. Tuller’s series on PACE, it is not correct to extrapolate from a subgroup selected from a group of Oxford patients to ME patients as a class. Further, the PACE recovery publication noted that CDC CFS criteria had been modified to require symptoms for just one week and that these modifications could result in inaccurate patient characterizations. The impact of PACE modifications to the ME criteria are unknown.

• White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M, PACE Trial Management Group. “Recovery from chronic fatigue syndrome after treatments given in the PACE trial.” Psychol Med. October 2013’ 43(10): 2226-2235. PMID: 23363640. http://dx.doi.org/10.1017/S0033291713000020

6 Smith MB, Haney E, McDonagh M, Pappas M, Daeges M, Wasson N, et al. “Treatment of Myalgic
Encephalomyelitis/Chronic Fatigue Syndrome: A Systematic Review for a National Institutes of Health Pathways to Prevention Workshop.” Ann Intern Med. 2015; 162: 841-850. http://dx.doi.org/10.7326/M15-0114

In the comments section, the authors stated, “Dr. Chu’s comment regarding the importance of analyzing data based on case definitions used for inclusion to trials is consistent with our approach. For example, in the trials of cognitive behavioral therapy (CBT) using the SF-36 physical function item as an outcome measure, the two studies using Oxford criteria indicated improvement, while the two using CDC criteria reported no improvement.”

7 UpToDate clinical guidelines, updated in July 2015, include:

o Gluckman, Stephen. “Clinical features and diagnosis of chronic fatigue syndrome (systemic exertion intolerance disease)” UpToDate. Deputy Editor Park, L. Last updated July 30, 2015. Literature review current through August 2015. http://www.uptodate.com/contents/clinical-features-and-diagnosis-of-chronic-fatiguesyndrome-systemic-exertion-intolerance-disease#H15

o Gluckman, Stephen. “Treatment of chronic fatigue syndrome (systemic exertion intolerance disease)” UpToDate. Deputy Editor Lee, P. Last updated July 30, 2015. Literature Review current through August 2015. http://www.uptodate.com/contents/treatment-of-chronic-fatigue-syndrome-systemic-exertion-intolerancedisease?source=see_link

8 Tucker, M. “IOM Gives Chronic Fatigue Syndrome a New Name and Definition” Medscape Multispecialty. February 10, 2015. http://www.medscape.com/viewarticle/839532

Dr. Clayton, IOM panel chair, noted, "The level of response is much more than would be seen with deconditioning," with reference to the belief voiced by some clinicians that physical abnormalities in these patients are merely a result of their lack of activity.