Skip to main content

MELA Burdened By Statistics Issue: Analyst

MELA may have exaggerated results from a study of its skin cancer detection device by swapping out statistics methods, an analyst claims.



) --

MELA Sciences


claimed success in a clinical trial of its experimental skin cancer detection device only by altering the statistical method used to analyze the data in violation of an agreement with U.S. regulators, charges an independent healthcare analyst in a report issued last week.

In response, MELA says it is confident in the statistical analysis of the data supporting MELAFind, disputing the report by

Biologic Equity Research

(BER), which has a sell rating on MELA based on the belief that MELAFind will not be approved in the U.S

MELAFind is a computer-enhanced imaging device that takes high-tech pictures of suspect moles and lesions to assist doctors in the early detection of melanoma, the deadliest form of skin cancer.

The U.S. Food and Drug Administration is convening an advisory panel on Nov. 18 to review the accuracy of MELAFind in diagnosing early-stage melanomas. The timing of the advisory panel is a bit unusual in that FDA told MELA last March that MELAFind was not approvable. MELA protested the FDA's rejection, submitted responses to the agency's concerns and demanded an advisory panel be held.

To this day, MELA has not disclosed fully the reasons why FDA issued a "not approvable " letter rejecting MELAfind. The BER report, however, relies on its own analysis to suggest that MELA struck out with FDA because the agency's medical device reviewers discovered the MELAFind pivotal study failed to reach statistical significance despite the company's claims to the contrary.

At Monday's close of $6.95, MELA shares are trading about where they were in March after the FDA's initial decision on MELAFind was announced.

MELA claims that a phase III study of MELAFind met its primary endpoint by detecting accurately 112 of 114 eligible melanomas for a "sensitivity" rate of 98%. The lower confidence bound of the sensitivity analysis was 95.1%, which met the FDA's standard for statistical significance in the study spelled out in a binding agreement with MELA, the company says.

The binding agreement between MELA and FDA covering the conduct of the MELAFind study required the company to analyze the data using a statistical method known as the "exact mid-P" test.

Clearing the 95% hurdle for the lower confidence bound using the mid-P test is important because that means if the phase III study were repeated, there would be a 5% chance or less that MELAFind's sensitivity to detect melanoma would be below 95%.

Any sensitivity less than 95% would compel FDA to reject MELAFind's approval.

In its report this week, the research shop BER says it was unable to reproduce MELA's positive results when it ran the MELAFind data independently using off-the-shelf statistics software.

"For the protocol-specified outcome group, the LCB

lower confidence bound using the exact mid-P method is 92.64% -- a far cry from the reported 95.1% lower confidence bound," the BER report states.

BER then ran the MELAFind data analysis again using a bunch of other statistics methods. What BER discovered was that MELAFind's publicly stated statistical results matched only when the "Wald normal approximation" method was used.

"The Wald method has been the subject of extensive criticism by statisticians for exaggerating results," the BER report states. Furthermore, use of the Wald method would violate the MELAFind study agreement signed between MELA and FDA.

MELA has also publicly announced results from two other sample groups in its phase III study of MELAFind. In both groups, MELA says MELAFind was highly accurate in detecting melanomas at sensitivity levels that exceeded the 95% lower confidence bound.

BER, running the same experiment, says it could not duplicate the positive, statistically significant results claimed by MELA from the MELAFind data using the FDA-mandated mid-P statistical test.

"We calculated the confidence interval for each of the three outcome groups using the exact mid-P method and the lower confidence bound for each of the three groups failed to meet the greater than 95% criteria for success," BER writes.

"In other words, this is a categorically and irretrievably failed trial if our assumptions are correct," the BER report concludes, referring to the MELAFind phase III study. BER has no investment in MELA, long or short.

MELA disputes the charges made by BER in its report but will not say if the company used an alternative statistical method to analyze the MELAFind data, nor will the company say if a disagreement over the conclusions reached by statistical analysis was the reason for FDA's "not approvable" letter.

"MELA has not disclosed any additional information beyond what's available on so I can't provided any further detail on the statistical analysis. Suffice it to say the company is confident in the statistical analysis of its data, particularly with respect to the confidence bounds on the sensitivity using the exact mid-P method," said MELA spokesman David Carey, in an email.

The U.S government's web site confirms that

MELA was required to use the mid-P statistics method

to analyze the MELAFind data. FDA does not make public the contents of not approvable letters and the agency doesn't comment on ongoing medical device reviews, so it's not possible to independently verify what statistical method was used by MELA.

MELA has touted the accuracy of MELAFind and the device's ability to detect melanoma skin cancers. In previous conference calls, MELA CEO Joseph Gulfo has said MELAFind's eventual approval and commercial launch would save lives and reduce the number of unnecessary biopsies. Dermatologists will find MELAFind particularly useful for early or suspect lesions or moles that are much harder to diagnose accurately and are often missed by traditional screening methods, Gulfo believes.

"MELAFind uses sophisticated algorithms and computer software to identify atypical lesions. These are things the human brain cannot do," said Gulfo on a June 29 conference call.


questions are raised about the accuracy and clinical relevance of MELAFind

by a full reading of both the study's final results and the way the study was designed:

  • MELAFind's 98% accuracy in detecting melanoma skin cancers (the device's sensitivity) was enhanced by the fact that all the lesions entered into the study were initially flagged as being possibly cancerous by dermatologists. This stacked the deck in MELAFind's favor, making it easier for the device to accurately diagnose melanoma.Despite the enriched pool of lesions entered into the phase III study, MELAFind still missed three melanomas, accurately diagnosing 172 of 175 lesions. This means that if MELAFind were the only tool used in these cases, three patients would have been told they were cancer free when in fact the lesions or moles on their bodies were melanoma.
  • MELA argues that MELAFind is still more accurate than the trained, experienced eyes and diagnostic ability of dermatologists. Yet the company has no data to prove that MELAFind's sensitivity is greater than that of dermatologists who participated in the study. The sensitivity of dermatologists could not be measured in the study, says MELA, because melanomas that dermatologists may have missed were not included in the phase III study.
  • Despite claims to the contrary, MELAFind does not significantly reduce the number of biopsies performed by dermatologists. In the phase III study, the biopsy ratio for MELAFind was 10.8 to 1, meaning that 10.8 biopsies were performed to catch one case of melanoma. For dermatologists in the study, the comparable biopsy rate was 11.3 to 1, which means using MELAFind reduced the number of biopsies performed by less than one. The clinical benefit of MELAFind is even more questionable when borderline lesions (the more difficult-to-diagnose cases) are added to melanomas in the analysis of the phase III data. Here, MELAFind's biopsy ratio was 7.6 to 1 compared to a biopsy ratio for dermatologists of 7.9 to 1 -- essentially equal.

--Written by Adam Feuerstein in Boston.

>To contact the writer of this article, click here:

Adam Feuerstein


>To follow the writer on Twitter, go to


>To submit a news tip, send an email to:


Adam Feuerstein writes regularly for In keeping with TSC's editorial policy, he doesn't own or short individual stocks, although he owns stock in He also doesn't invest in hedge funds or other private investment partnerships. Feuerstein appreciates your feedback;

click here

to send him an email.