Let's state the obvious: Warren Buffett is not a dumb man.

After all, he didn't earn the moniker "Oracle of Omaha" for nothing. But the 87-year-old was raked over the coals on Twitter this week after he said bitcoin will "definitely end badly."

To all those calling Buffett a dinosaur, a fraud or a failure for deciding to sit bitcoin out -- take a look in the mirror because you just might be the dumbest ones on Wall Street.

See, Buffett has seriously good reasoning. He's not getting in on bitcoin because he doesn't know enough about it. That's part of his investment strategy - the "circle of competence," as one source I spoke to put it.

This isn't the first time Buffett has faced such biting criticism. Buffett faced enormous backlash in the 1990s for his decision to stay away from internet stocks. Even then -- nearly 20 years ago -- he was criticized as old and outdated. But then the dotcom bubble popped, and Buffett was among those left standing tallest.

At the crux of it all is Buffett's investing philosophy: You don't have to swing at every pitch.

The key is to watch pitch after pitch go by. Wait for the right one for you - which might not be the best one for the next ball player on the batting order. Buffett is smart to ignore the calls to swing. His inaction when it comes to bitcoin is, after all, the smartest action to take.

As a source told me, "You're judged on the things you do better than the things you don't do."

Here's to you, Mr. Buffett. Don't let the hecklers from left field get you down.

Dig in to the Morning Jolt Archives:

Watch More with TheStreet: