NEW YORK ( TheStreet) -- "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." -- Warren BuffettAlthough his successful, multi-decade investing career has drawn followers, the roots of Warren Buffett's appeal can be traced back to more than just his ability to make money. Thanks to his extensive philanthropic work, optimistic outlook, and folksy, grandfather-like charm, the world famous investor has managed to construct a nearly bulletproof positive reputation. That's not to say that this reputation has not been tested. On the contrary, over the past few years, a number of the investor's actions have been greeted with ample criticism from market commentators and Buffett followers. For instance, around this time last year, fans of the investor were likely shaking their heads following his peculiar remarks before the Federal Crisis Inquiry Commission in regards to the actions of Moody's leading up to the financial crisis. Although the investor had been consistently reducing his exposure to the troubled ratings agency leading up to this testimony, Buffett was seen defending the firm, explaining that no one had seen the crisis coming. This wasn't the first time Buffett has defended a controversial company. Goldman Sachs ( GS) has received the Oracle of Omaha's praise on a number of occasions. In late 2009 the billionaire investor was even called upon to be an advisor for the firm's philanthropic program, "10,000 Small Businesses." More recently, Buffett drew flak following the unexpected resignation of former Berkshire Hathaway ( BRK.A) notable David Sokol. Although Sokol claimed to be stepping down to focus on his family's investments and his philanthropic efforts, the financial media immediately began to question Sokol's decision to purchase shares of Lubrizol ( LZ) in the months leading up to Berkshire Hathaway's acquisition. Buffett faced heat for his limited comments and initial defense of Sokol. It is important to note that in the weeks following, Buffett offered up a harsher criticism for Sokol's actions, and a Berkshire Hathaway committee found that the trades violated the company's code of ethics.