NEW YORK (TheStreet) -- Ben Bernanke can't refinance his mortgage, so how can you and I expect to do so?

After all, the former Federal Reserve Chairman reportedly earns $200,000 to $400,000 for a speech. Speaking at a conference in Chicago Thursday, Bernanke told the moderator, economist Mark Zandi that "just between the two of us...I recently tried to refinance my mortgage and I was unsuccessful in doing so." When the audience laughed, Bernanke said, "I'm not making that up," according to Bloomberg.

While amusing, the anecdote is indicative of a serious issue, and also rife with irony, said Richard Bove, analyst with Rafferty Capital Markets.

"If he can't get a mortgage, just think of all the tens of thousands of people who are trying to get mortgages right now who can't get them because of the stringency of the rules that he agreed to put in place," Bove said.

Indeed, there is plenty of data to back up the anecdotal evidence offered up by Bernanke that access to mortgage credit is too tight. Recent surveys by the Federal Reserve Bank of New York and Fannie Mae (FNMA) both show people are reluctant to buy homes for fear they won't qualify for a mortgage.

Bove argues over-regulation of big banks like Wells Fargo (WFC - Get Report) , Bank of America (BAC) , Citigroup (C) and JPMorgan Chase (JPM)  through new capital standards, for example, is directly responsible for the tightness of mortgage credit.

Jim Vogel, interest rate strategist with FTN Financial, believes it isn't over-regulation so much as the lack of clarity about what the rules are for mortgage finance that is the problem. In the wake of the crisis, for example, Fannie Mae and Freddie Mac (FMCC)  forced banks to buy back billions of dollars worth of home loans it says weren't properly underwritten.

"As long as mortgage originators know loans can return to them for years after they're sold to others, lending standards can't get easier," Vogel wrote in a research note published Wednesday.

In an interview, Vogel described the current housing finance system as having "sort of an ad hoc kind of feel." Referring to housing regulators, he said,"the people that have given you the assurance that we're done with the changes aren't necessarily empowered to say that."

Instead, Vogel believes it will take new legislation to fix housing finance. While bipartisan legislation passed the Senate Banking Committee earlier this year it lacked broad support and is widely seen as dead.

The situation is still not dire enough to force Congress's hand, Vogel contends.

"When people call their Congressperson and say 'why can't I get a mortgage?'--when that begins to happen is when you'll see Congress begin to move," he says.

Rafferty's Bove even believes the difficulties in mortgage finance could impact Wells Fargo shares, since the West Coast lender has more exposure to U.S. housing than more diversified peers like Bank of America, Citigroup and JPMorgan.

Referring to a recent court decision on Fannie and Freddie, Bove interprets it as saying the government sponsored enterprises should be wound down in accordance with the stated wishes of President Obama. With no clear plan for replacing the government sponsored mortgage giants, that means banks are less comfortable making loans, meaning they earn less money.

"Who's the biggest client of Fannie and Freddie in the United States? Wells Fargo," Bove said. "So it could be by association if the biggest buyer of mortgages in the United States goes away that Wells Fargo is going to have a problem in terms of where its going to sell its mortgages, or its going to be forced to keep its mortgages."

Follow @dan_freed
Read More: 
Bill Erbey Made $2.3B Off Your Underwater Mortgage

Disclosure: TheStreet's editorial policy prohibits staff editors, reporters and analysts from holding positions in any individual stocks.