NEW YORK ( TheStreet) -- New mortgage rules designed to create a more responsible mortgage market might end up hurting borrowers in states such as California and New York, where the price of real estate is expensive.

The Consumer Financial Protection Bureau recently issued rules that require banks to verify that the borrower has an ability to repay the loan by looking at criteria such as a borrowers' income, employment status, credit history and other debt obligations.

Banks will be offered greater legal protection if they make "qualified mortgages" -- loans that do not have excess upfront points and fees, have no toxic features such as interest-only loans, negative amortization and balloon payments, and where the borrower does not spend more than 43% of his income to pay down debt.

Analysts expect lenders to largely limit their lending to qualified mortgages to avoid legal liability.

While this is expected to prevent the kind of reckless and abusive lending practices of the past -- indeed, many banks have already tightened underwriting standards significantly -- the new rules could make it tougher for borrowers in high-end real estate markets.

According to a report by analysts at Deutsche Bank, the qualified mortgage rule will "significantly curtail the origination of prime jumbo loans."

Jumbo loans are those that are above the conforming limit of loans backed by Fannie Mae and Freddie Mac -- currently at $625,500.

The jumbo loan market has already shrunk, with most lenders sticking to mostly agency-backed mortgages. Those who do jumbo loans -- Redwood Trust ( RWT - Get Report) is a big player -- now write mostly prime loans.

Even after significant tightening of underwriting standards, more than 13% of jumbo loans originated since 2010 remain outside the parameters specified by the CFPB. Before 2010, about 75% of the jumbo loans were non-qualified mortgages.

Jumbo loans face a tougher test because, historically, many million-dollar homebuyers have not not depended on W2 incomes, the analysts note.

A lot of high-income borrowers with high credit scores tend to take "interest-only" loans for tax purposes. But interest-only loans are prohibited under the new rules.

The majority of jumbo loans prior to 2008 were highly levered, with debt well in excess of 43% of reported income. Since 2010, about 9% of prime loans originated have debt to income in excess of 43%.

Still, in markets such as California, New York and Connecticut, about 20% of jumbo loans originated in recent years would be considered non-qualified mortgages, according to the report.

"The adoption of qualified mortgages in January 2014 will limit credit availability for many buyers in these three States," the analysts wrote.

More broadly, the implementation of the rules could "further delay the recovery of the luxury housing market, which generally consists of homes that cost 1 million dollars or more. The luxury-housing sector is still struggling or even deteriorating despite continuous improvement in the overall U.S. housing market," the analysts said.

-- Written by Shanthi Bharatwaj in New York.

Disclosure: TheStreet's editorial policy prohibits staff editors and reporters from holding positions in any individual stocks.