The market turmoil couldn't come at a worse time for some boutique investment banks.

Since last month, investors have sold stocks en masse, fearing that

Federal Reserve

rate hikes will tip the economy into recession. The selloff has slammed brokerage stocks: The

Amex

broker-dealer index is down 17% in the last month, far outstripping a 7.7% loss in the

S&P 500

.

If the swoon has been tough up and down Wall Street, though, boutique investment banks have had it worst. Since May 10,

Thomas Wiesel Partners

( TWPG) is down 31%,

Greenhill

(GHL) - Get Report

is off 30% and

Jefferies

(JEF) - Get Report

is lower by 25%.

The selloff is complicating plans by several boutiques to go public. Cowen, a subsidiary of the investment banking arm of France's Societe Generale, filed Monday for a $235 million IPO. Evercore and Keefe Bruyette & Woods said earlier this year they would try to come public this year, too.

Until recently, investment banks were thriving, driven by solid economic growth and strong merger and IPO action. Boutiques saw strong gains across the financial sector and believed they could get good prices for their shares.

But given the action in the last month, some observers wonder if these banks haven't missed their window.

"Given the damage that we've incurred, it would not be the most beneficial time for an investment bank to come public," says Marc Pado, chief market strategist at Cantor Fitzgerald. "The environment you want is low to stable interest rates with economic growth looking to expand, especially for an investment bank because that is the market you do best in."

Rising interest rates and slowing economic growth don't make a great environment for banks of any sort.

Goldman Sachs

(GS) - Get Report

and

Lehman Brothers

( LEH) reported strong earnings earlier this week, but shares sold off as investors worry about the future. On the Lehman Brothers call, management said that it needed to "keep an eye" on its pipeline for deals in the coming quarter, as worsening market conditions could "delay or defer transactions."

In a market downturn, boutique banks are most disadvantaged. Many of the boutiques specialize in one area of business -- mergers or IPOs, for example. Adding to the weakness, many boutiques specialize in advising smaller clients. Those clients are more apt to pull deals than the large-cap companies, simply because they aren't quite as secure.

"So-called blue chip investment banks may be cushioned slightly because their clients tend to be leaders in their respective industries and thus less prone to contract their growth plans in the face of a recession," says Robert Bruner, dean of the Darden graduate school of business at the University of Virginia.

Meanwhile, the competition in the boutique banking sector is stiffening. Joseph Perella, a former

Morgan Stanley

(MS) - Get Report

investment banker, is expected to launch his boutique bank imminently. Other small banks, such as Rohatyn, are also competing for business.

As deal flow thins, the banks have a smaller fee pool to peck over. More competition dilutes the shrinking fees.

"The challenge for smaller firms is to survive the troughs," Bruner says.

For investors, the equation can be tricky. The key is understanding the bank's clients, Bruner says, because a bank's size doesn't necessarily translate into quality. Many smaller banks have top-tier clients. Evercore, for example, advises

AT&T

(T) - Get Report

,

General Motors

(GM) - Get Report

and

Viacom

(VIA) - Get Report

, among others. The firm's expertise might be enough to get it through a downturn.

"The key judgment for the investor in looking at investment banks' shares is the strength of the market franchise with respect to other banks," Bruner says. "Does it have a gilt-edged client base or not?"