Updated from 12:19 p.m. EDT

Warren Buffett's $5 billion investment in Goldman Sachs ( GS) is looking like a great deal for Berkshire Hathaway ( BRK-A) shareholders - and one Goldman isn't getting out of anytime soon.

Rochdale Securities analyst Richard Bove argued in a report on Saturday that Goldman should pay back Buffett's investment ahead of the government $10 billion preferred equity stake, as the terms of his deal are more costly to Goldman shareholders. However, a Goldman Sachs spokesman pointed out that under the terms of the government investment through the Troubled Asset Relief Program, companies must pay back the government before they pay back other shareholders. That includes preferred shareholders like Buffett.

On Sept. 23, Berkshire Hathaway made a $5 billion preferred equity investment in Goldman paying a 10% dividend. As part of the deal, Buffett's company has the right to buy an additional $5 billion in common shares at $115 per share, below the roughly $129 a share the company's stock was trading for on Monday.

The following month, the U.S. government took a $10 billion preferred stake in Goldman yielding 5% under the TARP's Capital Purchase Program (CPP).

Bove on Monday issued a new note clarifying his views.

"The point still stands," Bove wrote in an email message to TheStreet.com . "Also, the issue is the same -- do not dilute shareholders to eliminate TARP so that management can be paid more. Earn the money and then pay TARP back out of earnings."

First Niagara ( FNFG) on Monday said it was issuing new stock to repay the government's $184 million TARP investment and several other small banks also have sought to get out from under the government's thumb.

Goldman has indicated its desire to pay back the government as soon as possible. As its share price has more than doubled off last year's lows, speculation about an equity offering has been rampant.

Institutions that have government money have been subjected to intense public scrutiny, particularly over issues such as executive compensation. Furor over bonuses paid to executives as AIG ( AIG), which has received more than $70 billion, has been especially intense, but other large companies such as Morgan Stanley ( MS), Citigroup ( C) and Bank of America ( BAC) have also caught plenty of flack from Congress and the public.

Bove doesn't believe any company should be paying off any investors until it is certain that the crisis is over, but he also adds, "You shouldn't be diluting existing shareholders to pay off TARP so you can pay management more money."

Also over the weekend, Bove initiated coverage of JPMorgan Chase ( JPM) and Wells Fargo ( WFC) at Rochdale, where he moved earlier this year. He put a buy on both banks.