Any sensitivity less than 95% would compel FDA to reject MELAFind's approval. In its report this week, the research shop BER says it was unable to reproduce MELA's positive results when it ran the MELAFind data independently using off-the-shelf statistics software. "For the protocol-specified outcome group, the LCB
lower confidence bound using the exact mid-P method is 92.64% -- a far cry from the reported 95.1% lower confidence bound," the BER report states. BER then ran the MELAFind data analysis again using a bunch of other statistics methods. What BER discovered was that MELAFind's publicly stated statistical results matched only when the "Wald normal approximation" method was used. "The Wald method has been the subject of extensive criticism by statisticians for exaggerating results," the BER report states. Furthermore, use of the Wald method would violate the MELAFind study agreement signed between MELA and FDA. MELA has also publicly announced results from two other sample groups in its phase III study of MELAFind. In both groups, MELA says MELAFind was highly accurate in detecting melanomas at sensitivity levels that exceeded the 95% lower confidence bound. BER, running the same experiment, says it could not duplicate the positive, statistically significant results claimed by MELA from the MELAFind data using the FDA-mandated mid-P statistical test. "We calculated the confidence interval for each of the three outcome groups using the exact mid-P method and the lower confidence bound for each of the three groups failed to meet the greater than 95% criteria for success," BER writes. "In other words, this is a categorically and irretrievably failed trial if our assumptions are correct," the BER report concludes, referring to the MELAFind phase III study. BER has no investment in MELA, long or short. MELA disputes the charges made by BER in its report but will not say if the company used an alternative statistical method to analyze the MELAFind data, nor will the company say if a disagreement over the conclusions reached by statistical analysis was the reason for FDA's "not approvable" letter. "MELA has not disclosed any additional information beyond what's available on clinicaltrials.gov so I can't provided any further detail on the statistical analysis. Suffice it to say the company is confident in the statistical analysis of its data, particularly with respect to the confidence bounds on the sensitivity using the exact mid-P method," said MELA spokesman David Carey, in an email.
The U.S government's clinicaltrials.gov web site confirms that MELA was required to use the mid-P statistics method to analyze the MELAFind data. FDA does not make public the contents of not approvable letters and the agency doesn't comment on ongoing medical device reviews, so it's not possible to independently verify what statistical method was used by MELA. MELA has touted the accuracy of MELAFind and the device's ability to detect melanoma skin cancers. In previous conference calls, MELA CEO Joseph Gulfo has said MELAFind's eventual approval and commercial launch would save lives and reduce the number of unnecessary biopsies. Dermatologists will find MELAFind particularly useful for early or suspect lesions or moles that are much harder to diagnose accurately and are often missed by traditional screening methods, Gulfo believes. "MELAFind uses sophisticated algorithms and computer software to identify atypical lesions. These are things the human brain cannot do," said Gulfo on a June 29 conference call. But questions are raised about the accuracy and clinical relevance of MELAFind by a full reading of both the study's final results and the way the study was designed:
- MELAFind's 98% accuracy in detecting melanoma skin cancers (the device's sensitivity) was enhanced by the fact that all the lesions entered into the study were initially flagged as being possibly cancerous by dermatologists. This stacked the deck in MELAFind's favor, making it easier for the device to accurately diagnose melanoma. Despite the enriched pool of lesions entered into the phase III study, MELAFind still missed three melanomas, accurately diagnosing 172 of 175 lesions. This means that if MELAFind were the only tool used in these cases, three patients would have been told they were cancer free when in fact the lesions or moles on their bodies were melanoma.
- MELA argues that MELAFind is still more accurate than the trained, experienced eyes and diagnostic ability of dermatologists. Yet the company has no data to prove that MELAFind's sensitivity is greater than that of dermatologists who participated in the study. The sensitivity of dermatologists could not be measured in the study, says MELA, because melanomas that dermatologists may have missed were not included in the phase III study.
- Despite claims to the contrary, MELAFind does not significantly reduce the number of biopsies performed by dermatologists. In the phase III study, the biopsy ratio for MELAFind was 10.8 to 1, meaning that 10.8 biopsies were performed to catch one case of melanoma. For dermatologists in the study, the comparable biopsy rate was 11.3 to 1, which means using MELAFind reduced the number of biopsies performed by less than one. The clinical benefit of MELAFind is even more questionable when borderline lesions (the more difficult-to-diagnose cases) are added to melanomas in the analysis of the phase III data. Here, MELAFind's biopsy ratio was 7.6 to 1 compared to a biopsy ratio for dermatologists of 7.9 to 1 -- essentially equal.