It has not all been plain sailing, though. The Baltimore native has come under fire for his cost-cutting efforts, which have involved layoffs and shifting jobs to lower-cost parts of the world such as India.

"It's part of many a successful company," said Broadpoint AmTech's Marshall. "The sad state of affairs is that you're not going to have an optimal cost structure if you're running everything from the U.S."

IBM, which competes with Hewlett-Packard ( HPQ), Sun Microsystems ( JAVA) and Microsoft, raised its dividend this year, although Marshall feels that some of its competitors may offer more upside to investors.

Companies like EMC ( EMC), Brocade ( BRCD) and NetApp ( NTAP) are worth a look, according to the analyst. "I think there's more opportunities for finding growth at more attractive valuations than IBM," he said.

However, Palmisano is laying the groundwork for future growth. Although he is rarely seen on IBM's earnings calls, Palmisano is hard at work elsewhere, recently throwing his weight behind President Obama's health care IT agenda.

"I think he likes IBM CFO Mark Loughridge to focus on the investment community, so that he can focus on running the company and meeting with customers, business leaders, and government," he said.

Palmisano may not have scaled the heights of the music industry, but he is certainly earning a big reputation amongst tech's heavy hitters.

If you liked this article you might like

The Best Companies for Women

Hewlett Packard Enterprise Becomes the Latest Tech Titan to Slash Jobs

Sorry Elon Musk but Artificial Intelligence Grows Jobs: Domino's Pizza CEO

Ray Dalio Also Thinks AI Will Be a Killer Just Like Tesla's Elon Musk Does