Shares of the Armonk, N.Y., tech giant at last check were down 4.1% at $140.74. They're up nearly 12% year-to-date.
Chief Executive Arvind Krishna said in a statement that Whitehurst would continue as an adviser to him and all of IBM's senior management.
Whitehurst, 53, is the former CEO of Red Hat and Krishna said he "played a pivotal role" in the integration.
"In the almost three years since the acquisition was announced, Jim has been instrumental in articulating IBM’s strategy" and "ensuring that IBM and Red Hat work well together," Krishna said.
IBM in 2019 closed its $34 billion acquisition of Red Hat as it moved to reimagine its cloud business.
Krishna, who was head of the company's cloud computing business, was named CEO in January 2020. Krishna was also a principal architect of the company's acquisition of Red Hat.
In addition to Whitehurst, Bridget van Kralingen is stepping down as senior vice president of global markets. She will continue as senior vice president of special projects for a year and then retire.
Rob Thomas, who had been SVP of IBM's cloud and data platform, will succeed van Kralingen.
In that same month, the company said it had developed a microchip with 2-nanometer transistors, a move that could lead to better-performing phones and computers.
In April IBM reported first-quarter results that topped analyst estimates.