Practice Perfect 936
Residency In-Training Examinations: We Need More

Print

The American Board of Foot and Ankle Surgery (ABFAS) recently released the results of this year’s residency In-Training examinations (ITEs) for the first two resident years (PGY 1 and PGY 2). The PGY 3 residents must wait until November to receive their results. Residents will also take the American Board of Podiatric Medicine’s (ABPM) In-Training examination in February. I have always been a little ambivalent about these examinations and am aware of the multilayered reality of these tests. As a residency program director, I think we need more from our examination bodies.

On the surface, the ITEs are a standardized method to examine our residents as they proceed through their training. They take a didactic examination (both the ABFAS and ABPM) and a computer-based patient simulation (CBPS) through the ABFAS. In a very real sense, these exams are extensions of board examination series students take while in school through the National Board of Podiatric Medical Examiners (NBPME) and function as intermediaries along the process toward final board certification. As anyone who has been through the residency training process knows, it is not easy to create objective assessment methods. A lot of residency evaluations are based more on subjective observation and impressions by attendings than anything concrete and objective. Because of this issue, I’m highly supportive of the existence of the ITEs, and my program willingly participates and pays the expense yearly.

However, my problem is with how the test results are reported. Figure 1 shows the information enclosed in a report. Now, I don’t take issue with the categories or the comparison between the individual’s score and others in the PGY 1 and 2 years. I like that they break the sections down into parts (as opposed to a simplified report that just lists, for example, didactic and patient work-up sections).

Figure 1. Score report from the ABFAS In-Training Examination. Year and name removed for anonymity.

My complaint is what the report does not say. Within each examination category, it is impossible to know what the content of the actual question was. For example, looking at the “Diagnostic Studies/Medical Imaging” section, it is impossible to know if questions examined resident knowledge on MRI, radiography, scintigraphy, CT or other modalities. Was ultrasound included? Similarly, “Surgical Principles” is such a broad category that one could never evaluate a resident’s specific level of knowledge. Residents and program faculty are left with a vague gestalt. The CBPS section is no different. The “Physical Examination” category tells us nothing. Physical examination of what? What was the pathology?

The ABFAS publishes a guide to help residency directors, and in that guide they include information to try to help. Figure 2 below shows an excerpt from a table included in the 2018 guide. Since the table is too large to include here, I pulled the imaging section for our readers’ consideration.

Figure 2. Didactic imaging section from the ABFAS Guide for Residency Directors.

I applaud the ABFAS for attempting to break down the sections a bit more to provide information to residency personnel. However, it still doesn’t help. The problem is there are two forms of examinations, summative and formative, and the ITEs currently only provide summative information. The residents receive a “grade” (mostly a comparison against the mean).

Residency programs, though, need formative information. The real point of these examinations is to give our residents and programs feedback on how they are doing and empower us to make recommendations to study specific topics more (for the residents) and to show residencies where their overall deficiencies are. If all of the residents at a particular program score below the average on their imaging section, then that residency might want to adjust their imaging rotation or add imaging activities to their academics.

In the past, the ABPM released the actual examination after residents completed the ITE. I don’t know if they are still doing this, but as a director I loved knowing the content. We incorporated the test into our academics, reviewing specific questions during our academics. It also allowed us to direct residents to specific areas of study. In the long term, I don’t think this is a sustainable action for the boards because there is a finite number of questions and writing questions takes a lot of work. But there is a solution…

Tagging!
A reasonable compromise solution is to tag each of the questions on the examinations with information about the content. This simply requires the boards to become a bit more granular with what they are already doing. The podiatry schools are already doing this, so we know that it’s possible. I will state, here, that as someone who writes examination questions at the WesternU College of Podiatric Medicine, I don’t love tagging my questions. It’s laborious but far from impossible.

Let’s take an example based on the “Diagnostic studies/medical imaging” section from Figure 2. I write a question in which I’m testing a resident’s ability to diagnose osteoarthritis from a foot radiograph. The question requires them to understand that the radiographic signs of osteoarthritis are the following: asymmetrical joint space narrowing, eburnation, osteophytes, periarticular cysts, and intra-articular osseous bodies. The resident attempts to answer the question on the exam, and they get it wrong.

If this question were tagged, that tagging might look something like:

Foot Surgery > Didactic > Diagnostic Studies > Radiography > Pathology > Osteoarthritis

Each “>” indicates movement down the topic toward more specific information. There are any number of formats to this, and what I’ve suggested here is only one. As an examiner writes their questions, they would tag those questions down to the most specific topics. These tags could then be listed in a large table or spreadsheet as a reference for directors and residents.

In this specific example, the resident answered incorrectly. The ABFAS could then report “Incorrect” and the tagging tree listed above. They would not have to divulge any other information about the question, thereby maintaining their test bank integrity. This would allow a residency director to know that their resident is lacking knowledge about the radiography of osteoarthritis.

Since these exams are in general good ideas and mandatory for residency accreditation, it would be best if the boards could provide more specific formative feedback to empower both residencies and their residents to improve.

Best wishes.

Jarrod Shapiro, DPM
PRESENT Practice Perfect Editor
[email protected]