NEW YORK ( TheStreet) - For the first time, Warren Buffett appears concerned he will underperform the S&P 500 when it comes to his favorite way to peg the performance of his investing conglomerate, Berkshire Hathaway ( BRK.A). In Berkshire Hathaway's annual letter to shareholders, Buffett outlined why he is worried a rising stock market will put the firm's performance below that of the S&P 500 over a five-year stretch. Such a scenario would be the first in Berkshire's history, indicating that even the 'Oracle of Omaha' is having trouble keeping up with rising markets. Still, the lead to Buffett's self-deprecating annual letter may simply be a way for him to put a humble spin on a year when Berkshire's book value increased by $24.1 billion and the company's profit to shareholders nearly reached $15 billion, amid strong performance from new subsidiaries like Burlington Northern Santa Fe, Lubrizol and MidAmerican Energy and a solid year of investment gains from large holdings of Coca-Cola ( KO), Wells Fargo ( WFC) and American Express ( AXP). Berkshire's over 14% stock gain in 2012 also beat the S&P 500, a performance most investors would take as proof of a strong year. Buffett, in contrast, said in the letter to investors that he fears Berkshire's five-year book value growth rate may underperform the S&P 500 for the first time ever. In 2012, Berkshire's book value per share increased 14.4%, below gains for the S&P 500 in excess of 15%. The legendary value investor attributed his feared underperformance to rising markets, which may reflect expectations of continued economic recovery. "Our relative performance, however, is almost certain to be better when the market is down or flat. In years when the market is particularly strong, expect us to fall short," said Buffett. To that end, Buffett noted subpar book value gains were a result of Berkshire's operating subsidiaries. Meanwhile, the firm's investing portfolio significantly outperformed major stock indices. "They left me in the dust," Buffett wrote in an intentionally small-sized font, referring to Berkshire's two portfolio managers Todd Combs and Ted Weschler, who have taken an increasing role in common stock investments. A failure to shoot his so-called 'elephant guns' to bag a large acquisition was another disappointment for Buffett, who said at a Spring 2012 investor meeting that Berkshire had been close to a striking a $20 billion acquisition in December 2011. "The second disappointment in 2012 was my inability to make a major acquisition. I pursued a couple of elephants, but came up empty-handed." Notably, Buffett's statement indicates that in spite of contributing about $12 billion to the buyout of Heinz ( HNZ), in the company's $23 billion proposed takeover, Berkshire will be riding shotgun on the deal to private equity firm 3G Capital. While that deal "soaks up" much of Berkshire's 2012 earnings, Buffett indicated he will again hunt for a mega deal in 2013. " We still have plenty of cash and are generating more at a good clip. So it's back to work; Charlie and I have again donned our safari outfits and resumed our search for elephants," Buffett wrote. Berkshire-owned companies spent about $2.3 billion to make acquisitions, Buffett noted in his letter.