Applied Radiology

A conversation with Timothy Freer, MD and Michael Ulissey, MD

Drs. Timothy Freer and Michael Ulissey conducted the first large prospective clinical trial of the use of computer-aided detection (CAD) in screening mammography at Women's Diagnostic & Breast Health Center in Plano, TX. In their study of 12,860 women undergoing screening mammography, they have reported a 19.5% increase in cancers detected. Dr. Freer presented an abstract of their results at the Radiological Society of North America meeting in November 2000. ¹ Their complete study results were published in the September 2001 issue of Radiology . ²

While the study was designed primarily to measure the impact of CAD on radiologists' detection rate for breast cancer, additional data were collected to measure the collateral effects of using CAD, such as its effect on recall and biopsy rates. In a conversation with Applied Radiology , Drs. Freer and Ulissey discussed their results and their insights into the use of CAD for screening mammography.

The promise of CAD

AR: What role do you think CAD will play in radiology practices performing mammography?

Freer: Given our experience, we are no longer comfortable reading mammograms without CAD. Let's face it, most radiologists reading screening mammography are out there alone. We know that double-reading mammograms increases cancer detection, but this remains the exception because of the economic realities of medicine in this country. It was certainly that way for us.

We now know that we can reduce mortality from breast cancer by 63% in women who are getting their annual mammogram. ³ Nonetheless, 21% of the time, we overlook the extremely subtle signs of breast cancer and miss the opportunity for the earliest possible detection. Despite advances in technology, the false-negative rate of screening mammography is still far too high. No matter how experienced and diligent the radiologist is, the accuracy of mammography interpretation is limited by human perception, and computer-aided detection has the potential to eliminate the majority of these oversights.

AR: Which factors led you to purchase a CAD system for your facility?

Freer: The results of the premarket analysis of CAD were intriguing. They documented that CAD had the potential to "mark" mammographic signs of cancer initially missed by the radiologist, which should reduce the false-negative rate of mammography. The study by Dr. Warren Burhenne et al ⁴ reported that CAD marked the missed findings in 77% of false-negative prior mammograms. It's hard to ignore data like that. Most so-called "misses" are the result of a failure to perceive subtle changes on the mammogram. If these are pointed out to the radiologist, such a "miss" is less likely to occur, and CAD clearly had the ability to mark 75% to 80% of these lesions.

AR: What prompted you to conduct this study?

Freer: Frankly, we wanted to know if we'd made a good investment. A "potential" is exactly that. We needed to know just how well CAD actually performed in a clinical setting like ours, and what, if any, were the downside effects of its use.

The previously published reports documented that CAD had the potential to reduce the false-negative rate. But there was no data demonstrating the effect of the use of CAD on radiologists' sensitivity for detecting clinically occult breast cancers, on the recall rate, or on the outcome of the patients who were recalled. Each of these factors required prospective analysis to evaluate the clinical effectiveness of CAD for screening mammography. Our study was designed to assess exactly that.

AR: Your article mentions that you waited 75 days after the system was installed before beginning data collection for your study, in order to become familiar with the system and develop the procedures for using CAD. Can you comment on your experiences during this introductory period?

Ulissey: We mainly noted that CAD marked a lot of things. As it turns out, this is an important attribute of CAD. We learned not to obsess on any one mark, but rather to re-inspect any marked area to ensure we had not overlooked an important finding.

How CAD is used

AR: How does CAD analysis work?

Freer: The CAD system consists of two units: the processing unit and the display unit. The processing unit digitizes and analyzes the film images, and the display unit displays digital images of the mammogram on two monitors on the mammography autoviewer. The digital images are electronically linked by means of a bar code to the panels where the actual film images are mounted and are displayed by pressing a button on the autoviewer control panel (figure 1). Each digital image will display the marks, if any, that indicate areas where the detection algorithm recognizes a pattern that warrants closer evaluation by the radiologist. Two types of marks are used: an asterisk, which indicates a pattern suggestive of a mass or area of architectural distortion; and a solid triangle, which indicates an area of clustered bright spots suggestive of microcalcifications (figure 2).

AR: During your study, how did you use CAD for mammography screening?

Freer: How we used CAD in the mammography interpretation is particularly important. Every mammogram must first be read in the usual manner. Over our study year (February 2, 1998 to February 1, 1999), 12,860 women underwent routine screening mammography. When possible, each mammogram was compared with the patient's oldest prior mammogram of comparable quality or the most recent abnormal mammogram. In this interpretation, the radiologist must make a primary decision: either the study is "normal" or there are potential abnormalities that require further investigation. Only after this initial interpretation should CAD analysis be engaged, the mammogram re-reviewed, and any change in the radiologists' decision made.

Our study protocol required that once we re-evaluated an area marked by CAD, we could recall the patient or request prior mammograms to review for a potential abnormality not initially detected by the radiologist. However, the failure of the CAD system to mark a potential abnormality originally detected by the radiologist could not change the radiologist's decision to act on that finding.

This is a crucial point. A CAD analysis must never dissuade a radiologist from recalling a patient for further evaluation even if it has failed to "mark" an area that the radiologist believes is potentially abnormal. It can only be allowed to initiate a recall or request for prior films, by pointing out an area that the radiologist initially overlooked and, on close inspection, clearly merits further evaluation. The radiologist must be the final arbiter on the merits of any finding, regardless of how it was detected.

Results

Marking specificity

AR: How specific were the results of the CAD interpretation of the mammograms?

Freer: As Dr. Ulissey pointed out, the extremely low specificity of any one "mark" was considered an important attribute of the detection algorithm. Nearly 98% of the marks applied by CAD in this study were ultimately dismissed as unimportant by the interpreting radiologists.

Only 2.4% of the marks applied by CAD were recalled for diagnostic evaluation, and most of those marks were on findings already noted by the radiologist. The remaining marks were dismissed quickly and easily. This may sound like a weakness of the system, but it really affords us a two-fold advantage. First, the value of any detection system lies primarily in its sensitivity. In most systems, this requires a trade-off in specificity; the lower the specificity, the more confidence we had in the likelihood that most of the important lesions would be marked. Secondly, the low marking specificity prevented us from obsessing on any CAD mark that lacked visible mammographic findings, therefore not influencing our judgment. It did what it was designed to do--that is, to briefly focus our attention on an area of the image where a subtle cancer might have been overlooked.

Sensitivity for malignancy

AR: How did the results of the radiologists' interpretations compare with the results of the CAD system?

Freer: The radiologist and the CAD system were statistically equal in their ability to detect mammographic signs of malignancy. Of the 49 malignancies detected during the study period, the radiologists detected 41 lesions and missed 8 lesions prompted by the CAD system, increasing their sensitivity by 19.5%. The CAD system, in turn, detected 40 lesions, and failed to mark 9 lesions detected by the radiologist. More importantly, together, the radiologist and CAD were 20% better than either was alone.

Seven of the 8 lesions initially missed by the radiologist presented as clustered microcalcifications. All 9 malignancies that CAD failed to mark presented as masses.

While we understand that radiologists will be interested in this type of comparison data, we have to emphasize that current CAD systems are not designed to act independently. The focus should be placed on the performance of the radiologist when using CAD, not on how either performed alone.

Ulissey: It seems clear to us that more refinement of the mass-detection algorithms is needed, but we were concerned by the predominance of microcalcifications missed by the radiologist. Many radiologists may be overconfident in their ability to detect microcalcifications. They can be very obscure and easily overlooked.

Freer: This was certainly an eye-opener. In our study, 49% of the actionable findings initially detected by CAD were clustered microcalcifications. That may draw a number of critical comments. To those who suggest that the early detection of such lesions may have little impact on mortality reduction, we hasten to point out that 3 of the 7 missed lesions presenting as microcalcifications were moderate to high-grade ductal carcinoma in situ (DCIS) and 2 were invasive carcinoma. We all have increasing respect for the lethality of higher-grade DCIS, particularly comedo types, and for the fact that early invasive lesions may present as microcalcifications without a visible mass.

Effect on recall rate

AR: What effect did the CAD interpretation have on your recall rate?

Ulissey: It is important to understand that CAD, as used in our practice, can only increase the recall rate. This is because we did not allow the lack of a CAD mark on a finding initially considered worrisome to change our recommendation for further evaluation. CAD must never be used to help determine the importance of a mammographic finding; that job belongs to the radiologist. However, an increase in the recall rate is not counter-productive if there is a commensurate increase in cancers detected.

Freer: As Dr. Ulissey implied, if CAD had not led to an increased recall rate, it would mean that the radiologist had been dissuaded from recalling a patient because CAD failed to mark the radiologist's finding. Given the proportional improvement in cancer detection (of 19.5%), we believe that the 20% increase in our recall rate (from 6.5% to 7.7%) was quite acceptable.

Effect on biopsies

AR: With this increased recall rate, what change, if any, did you find in the rate of biopsies being performed?

Freer: With CAD, the number of biopsies performed in the screening program also increased by 20%. To measure the importance of this change, we compared the positive predictive value (PPV) for biopsy for both groups. The PPV for biopsies of lesions detected by the radiologist and those detected only after review of the CAD prompts were both 38%, so no impact on overall PPV for biopsy was observed.

This might have been predicted, since the decision to biopsy was based only on the merits of the lesion and not on the manner in which it was detected. Frankly, we seldom knew by which means the lesion was detected at the time of subsequent work-up. Therefore, if the PPV were to have fallen, we would have to surmise that there was something in the nature of the lesions prompted by CAD that would have led to a disproportionate number of negative biopsies, and we simply didn't observe this.

Effect on tumor stage

AR: What did you discover about the lesions initially detected by CAD?

Freer: The most encouraging effects of CAD were a 19.5% increase in the cancer detection rate, but we also measured an increase in the proportion of malignancies detected at an early stage. All 8 of the additional cancers detected by CAD were stage 0 or 1. It seems intuitive to me the cancers we were missing were likely to be early-stage lesions, and this notion is supported by our data.

Economic considerations

AR: With the current economic climate and the low reimbursement rate for screening mammography, the financial pressures on mammography centers continue to increase. How can radiologists justify the cost of a CAD system?

Freer: Certainly, the cost of current CAD systems is substantial, which is an important factor. Screening volume becomes the key element in minimizing its cost-per-patient and the necessary reimbursement to support its use. In our first 15 months of owning the CAD system, my colleagues and I bore the entire expense of the system. After completing our study, we recognized the benefit of CAD and felt obligated to provide it to as many of our patients as possible, but this required the purchase of a second system, which we simply couldn't afford. We were left with little choice but to enlist the support of our patients.

So, we offered the service of interpretation with CAD to our non-Medicare patients for a small surcharge. Medicare patients were given the service free-of-charge. The patients responded enthusiastically, and 80% of the screening patients consistently elected the service.

We were very gratified by the response of our patients. Nonetheless, we all look forward to the day when we can provide this service without asking for their support.

AR: There has been a good deal of coverage in the industry on the need for increased Medicare reimbursement for screening mammograms. Where does reimbursement for CAD interpretation stand?

Freer: We have had some good news. Earlier this year, the Health Care Finance Administration (HCFA) announced a Medicare reimbursement for CAD of $15 and is scheduled to increase the rate to approximately $19.50 on January 1, 2002. At that level, CAD should be financially feasible to large- and moderate-volume centers. Third-party payors are also beginning to take notice of the technology, and we hope that Medicare will again prove a reliable bellwether for private insurance companies. Further, the American College of Radiology recently announced that the AMA has approved a CPT I add-on code for CAD when used with screening mammography, effective 2002, which should make it easier to obtain private insurance coverage.

Medicolegal considerations

AR: Current statistics confirm that breast cancer cases are the leading cause of malpractice suits, especially claims of missed or late diagnosis. ^5,6 Berlin ⁷ has pointed out that many potential legal issues surrounding the use of CAD have yet to be clearly addressed. The use of CAD in screening mammography has not yet faced any legal challenges, but clearly many mammographers must be concerned about their potential medicolegal exposure. What can you suggest?

Freer: I must admit that it was one of my first concerns. What might the legal consequences be of failing to act on a potential abnormality marked by CAD? We hope that data such as ours will be useful in defending the position that a potential "missed lesion" can be judged only on its radiographic merits, and not on whether or not it was marked by a computer. For myself, I think the small malpractice risk that the use of CAD might pose is outweighed many times by the advantage of improved cancer detection. In addition, one can argue that by using a CAD system at your breast center, you are doing your best to reduce your possible oversights.

Future of CAD

AR: Since the FDA approved the first CAD system (R2 Technology, Inc., Los Altos, CA), more than 200 CAD systems have been installed in the United States. What do you believe the future of CAD will offer?

Freer: We think that the number of installed systems will increase rapidly, particularly if adequate reimbursement can be obtained. We really need to study several hundred thousand women to get the statistical power necessary to validate or refute the findings of our study. If future studies also confirm improved detection, CAD will prove to be a true technological leap and, perhaps, the greatest single advance in breast cancer detection in the last 20 years. *