Schloss: Surgeon Scorecard and the Fallacy of Aggregated Administrative Data

Edward J. Schloss, MD reviews the recent controversy over ProPublica's recent sensationalized public reporting of administratively-collected surgical mortality and readmission data:

"Some have argued that it was important to get this data out for public review, despite it’s limitations. I respectfully disagree. I subscribe to the belief that bad data is worse than no data. Certainly the scientific literature is replete with examples that prove this correct.

So is Surgeon Scorecard bad data? Strong words, but I say yes. This analysis was a great idea, but it fails to deliver on its goals. The data and methodology both have significant flaws. I say that from the perspective of a working clinician and clinical researcher with over 20 years experience, but I’d like to see a higher level of review. This project is as much science as it is journalism.  Surgeon Scorecard should be peer reviewed and critically discussed as would any scientific outcomes study. As I suggested to ProPublica, we need to kick the tires."

His analysis is a "must-read" for it speaks to many of the major flaws of using poorly collected and analyzed Big Data to improve medical care.

-Wes

Data Plan Health Care Shows Promise

CHICAGO - Citing mounting health care costs, electronic note bloat, and concerns with the quality and quality of Big Data, IBM, Apple, and EPIC Systems recently announced a new initiative to totally revamp US health care by offering health care services by data plan. The health care initiative was recently discovered in a little known section of the Patient Protection and Affordable Care Act (ACA) that changed portions of the U.S. Tax Code.

Under the new system, the brainchild of prominent Chicago physician-turned-health care entrepreneur Henry Throckmorton, MD, patients will purchase an initial 250 megabytes of data space on the EMR for all their health needs for $250 per month.  “It’s cheaper than most current cell phone service," Throckmorton explained. "When patients exceed their data allotment, health care ceases until patients purchase an additional data storage plan." Expansion data plans come in Bronze (250 M Bytes), Silver (500 Mbytes), Gold (5 G Byte), and Platinum (10 GB) storage increments.

Rollover plans for family members are also offered for those nearing the end of life.

“Health care systems that promise to limit the use of macros, dot phrases and cut-and-paste tactices have a real competetive advantage over competitors insensitive to the patient's data needs!” Throckmorton explained. "This system finally puts health care incentives in the right place.”

But Roger Wilco, spokesperson for America’s Health Insurance Plans (AHIP), the national trade association representing the health insurance industry, seemed less enthusiastic. "This is preposterous! Who do these flowery internet types think they are? Don't they realize there are advantages to more middle men in health care? How are we supposed to get our cut of the money?"

Dr. I.P. Knightly of Urocare Health System in Beaverton, New York, seemed less concerned about the middlemen and appreciated the improvements he's seen in patient care:  “Because I document everything on the EMR, including phone calls, results and work schedules, patient are less likely to call so I get a good night’s sleep!”

Nursing and medical students seem torn, however. While some see benefits to shorter notes, some like Tim Allen, MD, a hard-working fourth-year medical student from Roanoke, VA, sees other challenges “I’m still trying to understand ortho notes that no longer contain the critical information fields like the patient’s full name, VIP status, research status, and a complete review of systems. How's a guy supposed to understand what ‘Silt @ t/s/s/sp/dp’ means?”

Market analyst Rebecca Solomon of Lock, Stock and Barrel Equity Partners noted "Apple, IBM and EPIC are quickly gaining market share from more conventional insurance policies. The concept has also resonated with the Department of Health and Human Services because of the cost savings seen from fewer data-hungry imaging studies being ordered."

Mobile partnerships with AT&T, Verizon, and T-mobile are planned in the next fiscal year.

*  *  * 

 -Wes

P.S.: If you thought this press release might be real, even for a second, consider why.

The FDA Azithromycin Softshoe and Why It Matters

This week, a Danish study by Svanstrom and colleagues appeared in the New England Journal of Medicine that failed to show significant cardiovascular risk to azithromycin compared to other antibiotics in the Danish national health care system.  This report was in direct contradiction to an earlier report from Ray, et al. published in the New England Journal of Medicine (and widely hailed in press) that suggested a significant risk of cardiovascular death attributed to azithromycin compared to other antibiotics in a Tennesse Medicaid population - and a report that the FDA used to justify the wide dissemination of a Drug Safety Communication for azithromycin.

While I have been quite critical of the conclusions drawn by Ray, et al. before, I was even more surprised by the Drug Safety Communication for azithromycin issued by the FDA that basically said: "any time a patient has a bunch of conditions that are known to cause prolonged QT interval, be careful prescribing azithromycin."  Such warnings, of course, should apply to any drug that could cause prolongation of QT interval, but for some unclear reason, the FDA felt they had to act in the case of azithromycin.

Now it seems the FDA doing a softshoe on their earlier warning regarding azithromycin, trying to justify their warning that was based on retrospective data-mining techniques drawn from diagnosis and treatment codes, not randomized trials. 

In a perspective piece from Andrew D. Mosholder et al, from the FDA that accompanied the report from Svanstrom et al, the authors acknowedge the many limitations of the original study by Ray et al.:

"The study by Ray et al. has limitations that are intrinsic to obervational, nonrandomized clinical studies.  In particular, nonrandomized studies cannot exclude the possibility that patients receiving a drug under evaluation differ from the control patients in some important but undetected way, causing bias in the results. Such confounding may bias comparisons not only between patients receiving antibacterial drugs and those receiving no antibacterials but also between patients receiving different antibacterials."

Yet in the very next paragraph of the FDA member's perspective piece, they persist in their indifference to their prior statement regarding the bias inherent to Ray et al.'s report and decide to focus on timing:

"Despite these caveats, the results presented by Ray et al. warrant serious attention.  A chief strength of the study is the time-limited pattern of the risk: the azithromycin-associated increase in rates of death from any cause and from cardiovascular causes spanned days 1 through 5, reflecting the typical 5-day duration of azithromycin administration (e.g., Zithromax Z-pak)."

The FDA authors refuse to believe that there could be any other confounding variables that might have occurred in the first five days of hospitalization that lead to the increased cardiovascular risk seen in their azithromycin-treated group.  It is more important, I guess, to support retrospective data-mining and statistical data manipulation of diagnosis and procedure codes.  This, you see, they must support.  Big Data in our new era of expanded codes and computers is to be our new saving grace, you see.

Fortunately, I think the FDA folks mean well: they give us salient advice in their final paragraphs of their perspective piece explaining their mistake.  Things like "Clinicians must consider the arrhythmogenic potential not only of azithromycin but also of potential alternative antibacterial drugs." and "The risks and benefits of antibacterial therapy should be considered in prescribing decisions." 

No kidding.

But we will not see a black box warning retracted - that would be too obvious.  Nor will we see extensive news coverage about these new findings regarding this antibiotic's cardiovascular safety - that doesn't grab the main stream media's attention. 

What we have, instead, is something much more concerning to me when the members of our own FDA make claims based on poor data: irrelevance. 

Publishing warnings on QT prolongation that should apply to any drug administered to the patient that is older, bradycardic, with low potassium or magnesium levels, or already on known drugs that can cause prolonged QT - generates irrelevant noise for doctors.  Like the Boy Who Cried Wolf, we risk compromising the FDA's relevance to patient safety.   When they spend time on retrospective data-mining exercises that find risks in the neighborhood of 1 in 100,000, they risk failing to report important deaths related to new technologies with an incidence of at least 3 in 16,000 that reside in their own MAUDE database.

In today's internet age, doctors don't need any more Big Data-driven safety noise, we need razor-sharp analytic safety focus.

-Wes

Judgment versus Big Data

Knowledge in Numbers?

Decisions in medicine are supposed to rest on concrete obervations and hard evidence. 

Often, hard evidence does not exist or when it does, it isn't used.  Why is this? 

Concrete observations, too, are increasingly missed as we stare at computer screens longer and patients less.  Yet we persist. Why?

This is our reality now; our evolving medical world.

But if we stop and think about it, medicine, by definition, is a world of technological faults, systemic frailties, and human inadequacies.  We are convinced we know how a patient dies, for instance, thanks to the wonders of unprecedented imaging capabilities but stand slack jawed when an all-too-underperformed autopsy discloses a surprise cause of death that was completely missed by all.

And our answer to these inadequacies?  Stop doing autopsies.  Even though autopsies have consistently shown that one in four deaths occurs from an unexpected outcome or complication of care.

Why did we stop doing them?  Let me count the reasons: we are human, you see.

History repeats. 

Increasingly we are foregoing clinical judgment and intuition in favor of "Big Data" to make decisions.  We construct 70-page Appropriate Use Criteria for ICD documents that cover (really) just a few special clinical circumstances for patients, as if the authors ever really know a patient's clinical circumstance.  Ask yourself how good we are at predicting the day a person will take their last breath?  Like the weather, life is impossible to predict even when you have a billion data points or more.

Big Data and its certainty are our hottest trend in medicine and academics right now.  We know why this is: we love technology.  It is rational.  It is understandable.  It is linear.  We want, desperately, to understand and compartmentalize our human condition, to minimize its variability, so we can ration our resources logically.  But rather than acknowledging the limitation of such an approach, we forge ahead and create logic from dissociated databases with incomplete or empty data fields based on highly-selected patient populations to make our points.  Outliers are considered nothing more than acceptable loss rations.  We manipulate and massage the incomplete or erroneous data using statistics to make our points seem more valid.  Then, like the azithromycin folly, we extrapolate that data and transmit our firmly held beliefs through government agencies to the masses.  We feel good about our myopic analyses and are happy our academic salary was secured for another day.  In return, the importance of medical judgment, experience, and intuition to medicine are cast aside by our fervent belief that trials, databases, and data manipulation are always free from bias and the influence of greed.

More inadequacies.

But in the face of medical uncertainty, what other than judgment and intuition does a physician have - or a patient have, for that matter?  The real patient that sits before us demands an answer where, more likely than not, no real answer exists. Real concrete clinical challenges are rarely represented in a clinical trial or computer database. So we listen. We observe. We review data. Perhaps we get a second opinion. Patient judgment, life experiences, and intuitions are factored, too. Then we decide, together. Medical judgment and intuition are like that: not all luck, not all logic.


But now with Big Data, the new requirement for wellness and fitness is going to be for patients to keep proper symptoms that stay within the lines.  Symptoms and findings must fit new rubrics.  If they don't, your "caregiver" won't know how to treat you, the computer won't know how to treat you, and the rubric won't know how to treat you.  Who are you to say your symptoms are unique? Who are you to deserve a special look?  In the great cattle call of commoditized medicine created by Big Data, who do you think you are?   A liability risk?  Please, stay normative; align your symptoms with Big Data.  And be happy about it, dear patient, because the ends justifies the means.

Ironically, the folly of man has always been that we think we can have all the answers.  Perhaps we should stop for a moment and really think about what we're creating, courtesy of Big Data.

-Wes