At least two midsize banks appear to have fallen behind the curve on the subprime lending business.

Some on Wall Street are speculating that recent setbacks at

Huntington Bancshares

(HBAN) - Get Huntington Bancshares Incorporated (HBAN) Report

and

Commerce Bancorp

(CBH) - Get AllianzGI Convertible & Income 2024 Target Term Fund Report

are tied to investments with one hedge fund manager, Tom Brown of Second Curve Capital.

Brown's funds have taken huge paper losses this year on soured bets on companies that lend to customers with poor credit histories. Among his big holdings is

Accredited Home Lenders

( LEND), a San Diego-based mortgage banker that earlier this year agreed to sell most of its mortgage book to meet margin calls.

The news shows how risks tied to the swooning U.S. mortgage market could touch banks -- even ones that aren't known as big real estate plays.

Huntington, based in Columbus, Ohio, reported last month that first-quarter earnings slipped 8.4% from a year earlier, due in part to $8.5 million worth of equity investment losses. Commerce, based in Cherry Hill, N.J., posted $5 million in losses from equity investments when it reported earnings April 18, reflecting the "poor performance of several of these funds in the first quarter."

Neither bank specified where the losses came from, but "the speculation is that it was Tom Brown's fund," says one analyst who did not want to be identified.

Brown declined to comment for this article, saying in an email that "it would be a huge violation of our privacy policy to discuss who is or who is not a partner." A spokeswoman for Huntington declined to comment. Commerce did not return requests for comment.

Commerce, which runs 437 branches on the East Coast, said it has $40 million of investments in venture capital funds, mezzanine funds and bank stock funds. Huntington said it has invested $15 million since 2002 in "three financial services equity funds" in order to "improve competitive and market intelligence."

Huntington stressed that it remains "ahead in all three funds" despite recent setbacks. At the end of the quarter, those investments had a total value of $25.9 million. Huntington has more than 380 branches located in the Midwest.

Over the past few months, the subprime mortgage industry has screeched to a halt as loan delinquencies and defaults rose, cutting investors' appetite for mortgage-backed securities. Several lenders, including

New Century Financial

(NEWC)

, one of the largest subprime lenders, have filed for bankruptcy protections.

Brown's funds have dropped some 30% this year, according to a report in

Business Week

. Shares of Accredited Home have lost more than half their value this year. Last week, Brown upped his stake in Accredited to more than 11%, according to a filing with the

Securities and Exchange Commission

.

TheStreet Recommends

Second Curve also has stakes in

Capital One

(COF) - Get Capital One Financial Corporation Report

, which recently disclosed troubles in its newly acquired mortgage unit; in

NetBank

( NTBK), an online bank that exited the nonconforming lending business late last year; and in

First Marblehead

undefined

, a student loan servicing outsourcer.

First Marblehead's shares tumbled recently as investors resumed fretting that two of its biggest customers --

Bank of America

(BAC) - Get Bank of America Corp Report

and

J.P. Morgan Chase

(JPM) - Get JPMorgan Chase & Co. (JPM) Report

-- will jump ship once they complete their part in a $25 billion leveraged buyout of

Sallie Mae

(SLM) - Get SLM Corp Report

.

Observers say that while it might be normal for larger, more diversified banks to take risks in hedge fund investing, it is less common in smaller, deposit-focused banks.

Still, banks are dazzled by the potential rewards of hedge fund investing, says Tim Ghriskey of Solaris Asset Management.

"People see the returns and can't help themselves and want those returns for shareholders

and to boost the company's earnings," Ghriskey says. Solaris does not own positions in either Commerce or Huntington.

But Thomas Mitchell, an analyst at Miller Tabak who covers Huntington, wonders why it would want more exposure to an area that it already does business in: mortgage banking.

"Don't double up your bets by taking more risk in something you do business in," Mitchell says. "It would have been understandable if their loss was in Amaranth," a hedge fund that got blown up last year by losing natural gas trades.

"But to have a company that itself engages in mortgage business?" continues Mitchell. "To hire somebody else to invest in similar kinds of instruments perhaps with greater volatility doesn't make any sense at all."