Wednesday, July 09, 2014

Examining Maintenance of Certification Failure Rates

The American Board of Medical Specialties' (ABMS)  Maintenance of Certification® (MOC) program is marketed as "an ongoing process of education and assessment for certified physicians to improve practice performance."  After reviewing the recent failure rates of this program over the past five years, I worry the process is not about assuring physician quality, patient outcomes, or practice improvement, but rather as a means to ration the number of doctors eligible to earn Center for Medicare and Medicaid Services (CMS) payment incentives.

Background

As I've previously reviewed, the ABMS MOC program and the physician registry it creates has been written into our new health care law. To date, the MOC program serves as the sole measure of physician "quality" for the upcoming CMS value-based physician payment model that is to replace the current fee-for-service model in 2015.  To create an incentive for physicians to participate in the MOC program, CMS offered a 0.5% payment incentive to the Physician Quality Reporting System for physicians participating in the MOC program in 2014.  While this does not sound like much money, if we consider that physicians were paid $77 billion from CMS in 2012, this 0.5% represents approximately  $385 million dollars paid to doctors (or their employers).  Imagine the cost savings to the government if physicians were not eligible for such a payment.

Methods

I decided to evaluate the failures of "first time" MOC certification for all board certifications issued by the American Board of Internal Medicine from the pass rate data published online by the ABIM.  The number of failing physicians was calculated by subtracting the total number of doctors taking each examination from the number of doctors passing the test to arrive at the number who failed.  Non-integer values were rounded.  Next, I added up the total number of doctors who took the various ABIM MOC examinations each year and the total number who failed each year to generate an annual MOC percentage failure rate.  I calculated these values for 2009, 2010, 2011, 2012, and 2013.  I then applied a linear regression line comparing the total test takers and the total number of doctors failing the examination by year.  I then calculated the number of "certified" physicians each year as a "difference" of the total and failed physicians each year and applied a 2-period moving average trend line to these values.

Results

Here are the raw data assembled in a chart for your review: 

Year 2009 2010 2011 2012 2013
Total Test Takers (n) 8744 9574 10889 11524 12201
Number Failed (n) 861 1030 1357 1610 2138
Percent Failed 9.85% 10.76% 12.86% 13.97% 17.52%
Difference (n) 7883 8544 9532 9914 10063

Plotting these data shows the following trends:

MOC Failure Rate Trends (click to enlarge)

Discussion 

While the number of first-time MOC test takers grew each year studied, the failure rate also grew significantly. Is this because physicians were significantly less intelligent in 2013 than 2009?  Does this mean that board review courses run by each of our professional specialty societies are less relevant now than they were  despite their growing price?  Or might such a failure rate really be a way to "bend the cost curve" for health care delivery by covertly rationing the monies CMS pays physicians?  Each of these are fair questions that need to be answered honestly by CMS, the ABMS, the ABIM, and our professional specialty societies that collude with the ABMS as they run their various MOC board review courses. 

It goes without saying that test scoring methods and the raw responses of questions performed as part of the MOC process are shrouded in secrecy and serve no retrospective educational learning opportunity for doctors taking these examinations.  Doctors who take the MOC testing must also sign a statement that they understand that  divulging content in the examination will be met with harsh penalties including, but not limited to, possible revocation of hospital privileges or reporting to state medical licensing boards.  In return for this promise of secrecy, the ABMS and ABIM appears to operate in an environment that violates the trust of the public and those they test.  They do not explain their consistently higher failure rates seen year over year.  They do not mention the relationship they have to physician payments from government sources when physicians enroll in their MOC program.  Instead, they describe their process as "voluntary."  They espouse  the ethic of  "the need for public accountability and transparency," yet deliver none of these things themselves.  As such, it is clear that physician quality assurance or practice improvement is not the ABMS or ABIM's real mission for public good.

Rather, it appears from the MOC program failure trends above that the real reason for the ABMS MOC program is not only for self-enrichment, but to provide government cost savings without regard to the professional consequences to the many physicians they test and the patients ultimately affected by the loss of eligible care providers from their insurance panels. 


-Wes

6 comments:

Steve Parker, MD. said...

The only problem I see with your theory is that $385 is a drop in the ocean for the Federales.

I see the whole shebang as 1) a make-work boondoggle for parasitic bureaucrats (public and private), and 2) pay-offs for crony capitalists.

Anonymous said...

Patients don't care for MOC's either as they don't do squat for better care. Maybe they should get patient panels in and let US write about what is important to us.

Doing an excellent H&P, when to order bloodwork, how to interpret that, simple things are more important than the assinine stuff you see now.

Anonymous said...

YOu post is important but based only on first time takers of the test.
Did you see this graph from Sermo and the implications for those Recertifying:
https://app.sermo.com/posts/posts/224632?last_viewed_at=1404930316
Sermo | Sign in






Sermo | Sign in
If you are a physician licensed to practice in the United States (MD or DO), you are invited to connect with your colleagues on Sermo, the professional network started by physicians, for physicians.
View on app.sermo.com
Preview by Yahoo



I would suggest the text there indicates that the FIRST time testers are used as the index for pass/failing. This means it is prejudiced against the OLDER group of test takers (now would this be illegal due to the age discrimination?). If you have been continually exposed to the "latest statistics, etc" you will be better able to regurgitate that which is DESIRED (but also not necessarily true as it will fall in 2-5 years when the "new truth" is told).
It looks even worse on closer exam:
3413 total test takers with 65% pass means 1194 total failed of whole group
1867 of these were first time recerts and pass rate of 80% means 373 failed
This leaves 1546 "repeat or non-first time recertification testers" of which 821 failed!
This is a 53% failure rate among that group of second time recertification testers!
This means with every cycle of recertification you stand to get failed increasingly.
Seems like this is intended to weed out old guys/gals and make room for the young ones- AND YET THERE IS SUPPOSED TO BE A SHORTAGE OF PRIMARY CARE!
So it would seem with every "cycle" of 10 years, you are increasingly failing OLDER people, with indexing on the youngest only!

What are your thoughts to this?
I also actually requested CMS DATA on payments for PQRS-MOC in 2011 (the only year available and the results were published in Med economics which you may find useful as factual data:
http://medicaleconomics.modernmedicine.com/medical-economics/news/maintenance-certification-must-go-one-physicians-viewpoint?page=full
Maintenance of Certification must go: One physician's viewpoint


image





Maintenance of Certification must go: One physician's vi...
The Maintenance of Certification (MOC) program’s expense and time commitments continue to grow, producing greater complexity and more headaches f...
View on medicaleconomics....
Preview by Yahoo

MOC physician incentives
It is clear not all registered got their money and with 850,000 US docs, with a 2% cut in payments this will be a huge perhaps $2.5 billion cut in CMS payments equal to the "sequester" cuts which cause people to raise hell. This is why I think 2005-6 will be hallmark years for wake-up calls to physicians regarding the MOC extortion and government collusion and I hope results in large numbers dropping medicare and caid as providers-shifting care to Fauxicians.

Anonymous said...

Wes

Another great post. Even mediocre doctors do a better job when they take time. The MOC exam does not have provisions for this all important factor, which is quite possibly the most important aspect of who is getting good care. I can afford to take time IF my malpractice rates are not too high, if the administrators are not rationing out to me the time I may spend, and if I get a little sleep. Remember the sleep studies where the drowsy interns could not read EKGs. If MOC could provide for such considerations, it might gain credibility. Including questions on things a doc might see once or twice in a lifetime is not a good way to measure performance for the things he sees nearly every day.

Anonymous said...

You know...they decide who passes and fails.
So they can "FAIL" as many people as they want AND CHARGE another
$ 500 to retake the test.
No one knoes why they fail or even IF they fail.

Talk about a conflict of interest.

"You 20,000 people failed so pony up another $500 a piece".

Anonymous said...

I have taken the re-cert exam twice-in 2004 and 2014 ( passed each time) Am I considered first time taker or a repeater even though I passed each time.Does a " repeater " mean someone who has failed the test at least once or someone who has taken the exam more than once, even if they passed each time. That is not clear from the ABIM report