Wednesday, July 09, 2014

Examining Maintenance of Certification Failure Rates

The American Board of Medical Specialties' (ABMS)  Maintenance of Certification® (MOC) program is marketed as "an ongoing process of education and assessment for certified physicians to improve practice performance."  After reviewing the recent failure rates of this program over the past five years, I worry the process is not about assuring physician quality, patient outcomes, or practice improvement, but rather as a means to ration the number of doctors eligible to earn Center for Medicare and Medicaid Services (CMS) payment incentives.

Background

As I've previously reviewed, the ABMS MOC program and the physician registry it creates has been written into our new health care law. To date, the MOC program serves as the sole measure of physician "quality" for the upcoming CMS value-based physician payment model that is to replace the current fee-for-service model in 2015.  To create an incentive for physicians to participate in the MOC program, CMS offered a 0.5% payment incentive to the Physician Quality Reporting System for physicians participating in the MOC program in 2014.  While this does not sound like much money, if we consider that physicians were paid $77 billion from CMS in 2012, this 0.5% represents approximately  $385 million dollars paid to doctors (or their employers).  Imagine the cost savings to the government if physicians were not eligible for such a payment.

Methods

I decided to evaluate the failures of "first time" MOC certification for all board certifications issued by the American Board of Internal Medicine from the pass rate data published online by the ABIM.  The number of failing physicians was calculated by subtracting the total number of doctors taking each examination from the number of doctors passing the test to arrive at the number who failed.  Non-integer values were rounded.  Next, I added up the total number of doctors who took the various ABIM MOC examinations each year and the total number who failed each year to generate an annual MOC percentage failure rate.  I calculated these values for 2009, 2010, 2011, 2012, and 2013.  I then applied a linear regression line comparing the total test takers and the total number of doctors failing the examination by year.  I then calculated the number of "certified" physicians each year as a "difference" of the total and failed physicians each year and applied a 2-period moving average trend line to these values.

Results

Here are the raw data assembled in a chart for your review: 

Year 2009 2010 2011 2012 2013
Total Test Takers (n) 8744 9574 10889 11524 12201
Number Failed (n) 861 1030 1357 1610 2138
Percent Failed 9.85% 10.76% 12.86% 13.97% 17.52%
Difference (n) 7883 8544 9532 9914 10063

Plotting these data shows the following trends:

MOC Failure Rate Trends (click to enlarge)

Discussion 

While the number of first-time MOC test takers grew each year studied, the failure rate also grew significantly. Is this because physicians were significantly less intelligent in 2013 than 2009?  Does this mean that board review courses run by each of our professional specialty societies are less relevant now than they were  despite their growing price?  Or might such a failure rate really be a way to "bend the cost curve" for health care delivery by covertly rationing the monies CMS pays physicians?  Each of these are fair questions that need to be answered honestly by CMS, the ABMS, the ABIM, and our professional specialty societies that collude with the ABMS as they run their various MOC board review courses. 

It goes without saying that test scoring methods and the raw responses of questions performed as part of the MOC process are shrouded in secrecy and serve no retrospective educational learning opportunity for doctors taking these examinations.  Doctors who take the MOC testing must also sign a statement that they understand that  divulging content in the examination will be met with harsh penalties including, but not limited to, possible revocation of hospital privileges or reporting to state medical licensing boards.  In return for this promise of secrecy, the ABMS and ABIM appears to operate in an environment that violates the trust of the public and those they test.  They do not explain their consistently higher failure rates seen year over year.  They do not mention the relationship they have to physician payments from government sources when physicians enroll in their MOC program.  Instead, they describe their process as "voluntary."  They espouse  the ethic of  "the need for public accountability and transparency," yet deliver none of these things themselves.  As such, it is clear that physician quality assurance or practice improvement is not the ABMS or ABIM's real mission for public good.

Rather, it appears from the MOC program failure trends above that the real reason for the ABMS MOC program is not only for self-enrichment, but to provide government cost savings without regard to the professional consequences to the many physicians they test and the patients ultimately affected by the loss of eligible care providers from their insurance panels. 


-Wes

9 comments:

Steve Parker, MD. said...

The only problem I see with your theory is that $385 is a drop in the ocean for the Federales.

I see the whole shebang as 1) a make-work boondoggle for parasitic bureaucrats (public and private), and 2) pay-offs for crony capitalists.

Anonymous said...

Patients don't care for MOC's either as they don't do squat for better care. Maybe they should get patient panels in and let US write about what is important to us.

Doing an excellent H&P, when to order bloodwork, how to interpret that, simple things are more important than the assinine stuff you see now.

Anonymous said...

Wes

Another great post. Even mediocre doctors do a better job when they take time. The MOC exam does not have provisions for this all important factor, which is quite possibly the most important aspect of who is getting good care. I can afford to take time IF my malpractice rates are not too high, if the administrators are not rationing out to me the time I may spend, and if I get a little sleep. Remember the sleep studies where the drowsy interns could not read EKGs. If MOC could provide for such considerations, it might gain credibility. Including questions on things a doc might see once or twice in a lifetime is not a good way to measure performance for the things he sees nearly every day.

Anonymous said...

You know...they decide who passes and fails.
So they can "FAIL" as many people as they want AND CHARGE another
$ 500 to retake the test.
No one knoes why they fail or even IF they fail.

Talk about a conflict of interest.

"You 20,000 people failed so pony up another $500 a piece".

Anonymous said...

I have taken the re-cert exam twice-in 2004 and 2014 ( passed each time) Am I considered first time taker or a repeater even though I passed each time.Does a " repeater " mean someone who has failed the test at least once or someone who has taken the exam more than once, even if they passed each time. That is not clear from the ABIM report

Anonymous said...

I recently had an email discussion with one of the individuals at ABIM. Since the ABIM keeps track of each certified physician by their ABIM number I thought it was not unreasonable to ask them to parse through the data and let us know the pass/fail rate of those who are taking the recertification exam for the first time (approximately 10 years out of their IM residency) as opposed to those who are taking the recertification exam for the second time or re-re-certification (approximately 20 years out of residency) I have posted the thread below.....

Due to length It is posted in 3 separate comments.
There are 2 emails and 2 responses: Overall he did not actually address the issue...

Email # 1
From Address: Dr XXXXX
Of those that did not pass the recent Internal Medicine recertification exam (myself included) I am wondering how many of those individual were now getting re-certified for the second time (like myself) as opposed to getting re-certified for the first time. I suspect that those of us who are 20 years out from completing our residency would have a harder time with the exam.

My practice consists almost entirely of Allergy & Immunology. I found that the questions on the ABIM re-certifiaction exam did not pertain to what I would see as core material that all Internists/ IM sub-specialists should be aware of. Much of the exam covered hospital based issues as well as other minutia that even a general internist would not be exposed to on a regular basis. Perhaps there should be separate exams, for specialist vs Generalist. Also perhaps the scoring should be adjusted to take into consideration the number of years since completing initial certification.

I would appreciate a response at your earliest convenience. Included in this I would like to see the specific data as to whether those who passed/failed were 10 or 20 years out....

Anonymous said...

Response #1
On Jul 17, 2014, at 10:03 AM, Louis J. Grosso wrote:

Dear Dr. XXXXX

You’re email below was forwarded to me for reply.
Thank you for your e-mail regarding the content of the Maintenance of Certification (MOC) examination and requesting data comparing the numbers of physicians that passed/failed the MOC exam who were initially certified ten and twenty years ago. I understand your concerns about not passing the Internal Medicine MOC exam and will do my best to address those concerns here.

We are unable to provide the data you requested on the pass/fail rates based on when physicians were initially certified. ABIM only provides examination data that are publically available on the ABIM website. We receive numerous requests for data and information regarding our examinations and candidate population. Because of the resource commitment that would be necessary, fulfilling every request is not possible and it would not be fair to grant a request to some and not others.

With regard to your comments about the content included on the exam, please understand that ABIM spends a considerable amount of time and effort to ensure that examination content is carefully blueprinted to be comprehensive and relevant. Questions are written to reflect this blueprint and reviewed/revised two or three times by ABIM. After they are found acceptable but before they become part of the test, the questions receive yet another review by a larger group of practitioners. The goal of this process is to assure that the examination is thorough, relevant and fair test of medical knowledge and clinical judgment. Despite these efforts, we recognize that the test as a whole will not be equally relevant to all candidates, in part because of variations in practice patterns.

I also understand your desire for separate exams for generalists and subspecialties and adjusted scoring given your disappointment in not passing the examination. ABIM, in line with the requirements of the American Board of Medical Specialties, has established a common pathway to MOC for all candidates. One component of this process is passing the written examination with a common standard applied to all. This single pathway ensures that all candidates are treated fairly and that everyone faces exactly the same challenge. To offer an alternative based on a different standard or examination would be unfair to candidates who have already been successful and it would alter the meaning and credibility of the certificate.

I hope this is helpful in addressing your concerns.

Sincerely,
Louis Grosso

Vice President,
Knowledge Assessment – Production and Analysis
American Board of Internal Medcine
510 Walnut Street, Suite 1700
Philadelphia, PA 19106

Phone: 215-446-3538

Anonymous said...

Email #2 and response #2

Email #2

From: Dr XXXXXSent: Sunday, July 20, 2014 1:37 AM


To: Louis J. Grosso
Subject:
Re: External Request for Data –

Your response implies that my concern is a case of “sour grapes” I am more concerned with the overall trend. I do not NEED to be recertified in Internal Medicine in order to continue to practice Allergy/Immunology. I find it alarming that there was a 22% failure rate. You responded by telling me that: "ABIM only provides examination data that are publically available on the ABIM website” You are using circular logic, since of course you are only using data that is currently available on the website. The whole point of my question is that you SHOULD be including more detailed information and statistics on the website. Most of the “first time takers" I assume are only 10 years out from their initial certification. My guess is that those who are 20 years out probably have a higher failure rate. Knowing this would be valuable in that it would help those like myself decide whether or not it is worth the time and money to attempt to re-recertify after 20 or 30 years.

Response #2
Dear Dr XXXXX,

My email was not based on any assumptions about the intent of your questions. It was simply an attempt to respond to your queries.

I do understand your interest in seeing more detailed information. That being said, any finer breakdown of data needs to be conducted with appropriate rigor. Keep in mind that a candidate can elect to take an exam as early as the sixth year of a certification cycle. Consequently, there can be individuals who finished training from six to 20 years earlier. Although other measures of ability are key markers in the likelihood of success on the examination, there are a range of other factors that will contribute.

Email is not the best way to address your concerns. If you’d like to discuss this further I would be very happy to speak with you by telephone.

Respectfully,

Louis Grosso

Vice President,
Knowledge Assessment – Production and Analysis
American Board of Internal Medcine
510 Walnut Street, Suite 1700
Philadelphia, PA 19106

Phone: 215-446-3538

Anonymous said...

Looks like the overall failure rate for 2014 is again >17% if I have calculated correctly?