Rebuttal: The NEJM's Justification of Medical "Mystery Shoppers"

The proposed examination of access to primary care according to insurance status in nine representative states was largely derailed by physicians and other critics concerned about the potential for government invasion of physicians' privacy. They argued that less controversial survey methods would suffice or that additional studies of the well-known primary care shortage are a waste of public resources. I think these arguments are misguided.

- Karin Rhodes, MD NEJM, July 27, 2011 (10.1056/NEJMp1107779)

Fair enough, Dr. Rhodes. You certainly are entitled to your opinion.

But before I take on my rebuttal to your piece, let's both be clear on a separate issue: what is most misguided about your perspective piece in the New England Journal of Medicine was that comments were not allowed. If they were, the "physicians and other critics" could explain their aversion to these tactics.

So, let me be the first to state my position.

Covert, subversive tactics in research in an attempt to avoid bias carries the risk of introducing additional forms of bias. For instance, when a phone call is made to a doctor's office for a new patient appointment and the problem sufficienctly urgent that other real live patients are rescheduled to accommodate a mystery patient's needs, how, exactly are the affects and costs to the established patients compensated? How will those data be "counted" in your statistics when your one new patient's access if offset by the loss of two follow-up patient's access. Who will explain to those affected by these tactics why they were rescheduled? Will you?

You claim that "the study was intended to generate valid national estimates of primary care capacity before the anticipated expansion of private and public insurance to as many as 38 million currently uninsured Americans."

First, recall the problems with that "38 million" number. Wasn't that number "47 million uninsured not too long ago?" Right off the bat, we see how numbers can be spun in policy circles, Dr. Rhodes. Which leads to the most important question that remains unanswered regarding a study that uses these covert tactics: how will the data be used? Will the data (which most certainly are going to "discover" problems with access) be used to justify mandates to shorten office visits from 7.5 minutes per patient to 7 minutes per patient to improve access? Or might doctors be directed to see more patients that are not insured? Seriously. What policy directives can we expect from these data?

You justify the use of this deceptive practice saying "the use of masking and concealed allocation, widely endorsed for randomized, double-blind clinical trials, lends confidence to the interpretation of results." And yet in the circumstance of randomized, controlled trials, patients must sign informed consent to take part in such a study before they are randomized. Hardly a "mystery" process. Should patients and doctors of prospective clinics not be afforded the same respect who might be asked to take part in your study?

You also seem to feel that a sampling 18% of states (9 of 50) is adequate to formulate conclusions. I find this concerning. National policy development should have representation from all states affected, not a minority. To suggest that the concerns of states with relatively high congestion mirror those with more rural populations is certain to bias policy decisions going forward and, more likely than not, exclude the perspective of less populous states.

So these are just a few of my concerns. There are others. Please note that none of them even begin to address the privacy issues raised by "others." But given the flaws I've outlined, paired with the obvious shortage of physicians that we will encounter in 2014 when the full brunt of the Patient Protection and Affordable Care Act kicks in (not to mention our limited research funding these days), this study certainly does appear to squander our limited public resources. Must we spend our resources to become Masters of the Obvious?

No doubt others would like to share their views, so unlike the New England Journal of Medicine, I'll leave my comments open.

-Wes