) - For the first time, Warren Buffett appears concerned he will underperform the

S&P 500

when it comes to his favorite way to peg the performance of his investing conglomerate,

Berkshire Hathaway

(BRK.A) - Get Report


In Berkshire Hathaway's

annual letter

to shareholders, Buffett outlined why he is worried a rising stock market will put the firm's performance below that of the S&P 500 over a five-year stretch.

Such a scenario would be the first in Berkshire's history, indicating that even the 'Oracle of Omaha' is having trouble keeping up with rising markets.

Still, the lead to Buffett's self-deprecating

annual letter

may simply be a way for him to put a humble spin on a year when Berkshire's book value increased by $24.1 billion and the company's profit to shareholders nearly reached $15 billion, amid strong performance from new subsidiaries like

Burlington Northern Santa Fe




MidAmerican Energy

and a solid year of investment gains from large holdings of


(KO) - Get Report


Wells Fargo

(WFC) - Get Report


American Express

(AXP) - Get Report


Berkshire's over 14% stock gain in 2012 also beat the S&P 500, a performance most investors would take as proof of a strong year.

Buffett, in contrast, said in the letter to investors that he fears Berkshire's five-year book value growth rate may underperform the S&P 500 for the first time ever. In 2012, Berkshire's book value per share increased 14.4%, below gains for the S&P 500 in excess of 15%.

The legendary value investor attributed his feared underperformance to rising markets, which may reflect expectations of continued economic recovery.

"Our relative performance, however, is almost certain to be better when the market is down or flat. In years when the market is particularly strong, expect us to fall short," said Buffett.

To that end, Buffett noted subpar book value gains were a result of Berkshire's operating subsidiaries. Meanwhile, the firm's investing portfolio significantly outperformed major stock indices. "They left me in the dust," Buffett wrote in an intentionally small-sized font, referring to Berkshire's two portfolio managers Todd Combs and Ted Weschler, who have taken an increasing role in common stock investments.

A failure to shoot his so-called 'elephant guns' to bag a large acquisition was another disappointment for Buffett, who said at a Spring 2012 investor meeting that Berkshire had been close to a striking a $20 billion acquisition in December 2011.

"The second disappointment in 2012 was my inability to make a major acquisition. I pursued a couple of elephants, but came up empty-handed."

Notably, Buffett's statement indicates that in spite of contributing about $12 billion to the buyout of



, in the company's

$23 billion proposed takeover

, Berkshire will be riding shotgun on the deal to private equity firm

3G Capital


While that deal "soaks up" much of Berkshire's 2012 earnings, Buffett indicated he will again hunt for a mega deal in 2013. "

We still have plenty of cash and are generating more at a good clip. So it's back to work; Charlie and I have again donned our safari outfits and resumed our search for elephants," Buffett wrote.

Berkshire-owned companies spent about $2.3 billion to make acquisitions, Buffett noted in his letter.

Buffett also outlined how Berkshire Hathaway changed some of its investing principles, but kept others intact.

After buying back stock at a price of 120% of book value in late 2012, Buffett said on Friday he will use that ratio as a target to

buy back shares


"We originally said we would not pay more than 110% of book value, but that proved unrealistic. Therefore, we increased the limit to 120% in December when a large block became available at about 116% of book value."

While Berkshire raised the price at which it might consider buying back stock, succumbing to some shareholder demands, Buffett was not ready to commit to paying out a dividend, a policy that the 'Oracle' said has been pressed by many of his friends and investors.

Buffett indicated that Berkshire Hathaway will only pay a dividend if the company's book value growth continues to underperform overall markets - a real concern according to 2012 results.


We believe our assumptions about the book-value buildup and the market-price premium seem reasonable. If the prospects for either factor change materially for the worse, we will reexamine our actions," said Buffett.

Still, in the letter, Buffett outlined once more why dividends are a staple of many companies, including Berkshire's top four stock holdings, that also can create significant shareholder value.

Although Buffett said a rising market dampened Berkshire's relative performance in 2012, he nonetheless stood by a bullish stance on investing in America and the prospect of a growing economic recovery.

"There was a lot of hand-wringing last year among CEOs who cried 'uncertainty' when faced with capital allocation decisions... At Berkshire, we didn't share their fears, instead spending a record $9.8 billion on plant and equipment in 2012, about 88% of it in the United States," wrote Buffett, noting the figure was a new record.

Overall, the down note taken by Buffett in Berkshire's annual letter may be little more than a way for the 'Oracle of Omaha' to present a strong year for the investing conglomerate.

Berkshire Class A shares recently hit record highs above $152,000.00 a share in February.

Follow @antoinegara

-- Written by Antoine Gara in New York