NEW YORK (

TheStreet

) -- An official congressional report on Tuesday outlined what observers of the mortgage mess have been saying for weeks: The "robosigning" scandal and buyback bombs are cause for concern but those concerns have probably been overblown.

Perhaps the most telling conclusion of the 127-page

report by the Congressional Oversight Panel is its buyback-loss estimate: $52 billion, for the entire industry, spread out from 2009 through perhaps 2013.

Accounting for losses already incurred and reserves already set aside, that leaves just under $31 billion in losses over the next two or three years. Most of that cost will be borne by the country's largest servicers,

Bank of America

(BAC) - Get Report

,

JPMorgan Chase

(JPM) - Get Report

,

Wells Fargo

(WFC) - Get Report

and

Citigroup

(C) - Get Report

. While it isn't chump change, it's far less than the $74 billion or $375 billion figures that were being tossed around the market in recent weeks for BofA alone.

The buyback issue relates to claims from parties that own or guarantee mortgages that big banks securitized or service. Those parties claim that "representations and warranties" in purchasing contracts have been breached, which would require securitizers to buy back the debt.

Government-sponsored entities (GSEs)

Fannie Mae

(FNMA.OB)

and

Freddie Mac

(FMCC.OB)

have been the most successful in forcing banks to buy back souring mortgage bonds, but others are pursuing claims as well. Monoline insurers like

MBIA

(MBI) - Get Report

and private investors are pursuing litigation while other investors including

Blackrock

(BLK) - Get Report

, Pimco and the

Federal Reserve

have requested buybacks out of court.

"Losses stemming from mortgage put-backs are viewed as the biggest potential liability of the banking sector from the foreclosure crisis," the panel said in its report. But it also noted that "private-label investors do not benefit from the same degree of protection" as GSE contracts and that there are "logistical impediments to centralizing claims, in addition to the higher hurdles necessary to putback securities successfully to the banks."

In regards to the "robosigning" issue, the panel's didn't offer any solid conclusions about big banks' legal liability. But it did cast doubt on how far any court case might go.

Several lenders have faced allegations - and in some cases found evidence - that their employees had been improperly signing affidavits under oath without verifying borrowers' information. Bank of America, Ally Financial (formerly known as GMAC), JPMorgan, Wells Fargo and

PNC Financial Services

(PNC) - Get Report

, among others, have reviewed their foreclosure practices and found flaws in hundreds of thousands of cases.

The panel said it was "very unlikely" that any nationwide class-action lawsuit could be successfully pursued, due to variations in state law. Investigations by all 50 attorneys general may prove to be a more serious problem for the banks.

The panel offered a popular refrain to describe the potential outcome of those cases: "Uncertain," a phrase it used nearly two dozen times to describe various aspects of the mortgage mess.

"Allegations of "robo-signing" are deeply disturbing and have given rise to ongoing federal and state investigations," the panel said, concluding that, "at this point the ultimate implications remain unclear."

While the panel said it's "possible" that robosigners "at least are guilty of perjury," it concluded that even those cases would be difficult to prosecute successfully, because prosecutors would have to prove that employees intentionally and knowingly lied.

The panel outlined two potential scenarios: In the best-case, the robosigning scandal represents "mere clerical errors that can and will be resolved quickly"; in the worst-case, it reflects "extensive misbehavior" across the mortgage industry which could throw the housing market into a tizzy. In that scenario, it may be unclear who has legal property rights to millions of homes and the $7.6 trillion in mortgage-backed securities that stand behind them.

The panel suggested that regulators closely monitor the situation and conduct stress tests to ensure banks can handle such a doomsday situation.

"The risk is uncertain, but the danger is significant enough that Treasury and all other government agencies with a role to play in the mortgage market must focus on preventing another such shock," the report said.

-- Written by Lauren Tara LaCapra in New York

.

>To contact the writer of this article, click here:

Lauren Tara LaCapra

.

>To follow the writer on Twitter, go to

http://twitter.com/laurenlacapra

.

>To submit a news tip, send an email to:

tips@thestreet.com

.

Disclosure: TheStreet's editorial policy prohibits staff editors, reporters and analysts from holding positions in any individual stocks.