While Mark Cuban sees more value in a holding a "pet rock" than the yellow metal, Kevin O'Leary said it's a definite part of his portfolio.

O'Leary said gold is the only asset he owns that doesn't pay a dividend, but he sees the metal's role in his portfolio is an inflation hedge and a "buffer."

His comments come as gold has been range-bound since the start of the year. Comex June gold futures last traded at $1,341.10 an ounce, up almost 3% since January. The yellow metal has found recent support after the Federal Reserve announced last month that it only sees three rate hikes in 2018.

O'Leary's comments are also a sharp contrast to his guest at the "Three Sharks in a Castle," symposium, Mark Cuban, who sees zero value in owning gold as an investment.

"I would buy a pet rock before I buy gold," the billionaire owner of the Dallas Mavericks told Kitco News ahead of the event in Toronto.

But O'Leary's fondness for the physical metal does not translate to mining equities, noting that investors are better off holding physical gold than the mining stocks.

Gold miners have had a track record of bad management and poor cost control that have led to underperformance relative to physical gold, the chairman of O'Leary ETFs told Kitco News on the sidelines of the symposium. The Shark Tank investor himself maintains 5% of his portfolio in physical gold, using a combination of bullion and SPDR Gold Shares (NYSE: GLD), the world's largest gold-backed exchange-traded fund.

"The history of mining has been abysmal," he said. "The value of the commodity is whatever it is every day. I don't need to have a manager in the middle screwing up his capital cost allowance, not controlling his costs," he said.

Gold miners lagged behind physical gold in 2017, with the VanEck Vectors Gold Miners ETF (NYSE:GDX) up only 7.7% on the year while GLD climbed 12% in the same period.

Mining stocks were depressed last year despite many companies having reported lower all-in sustaining costs and more efficient production.

"I say to the [miners] you do what you have to do to get [gold] out of the ground. I'm not going to invest in that. When it arrives, I'll buy it, and that's served me very, very well over time," O'Leary explained.

Some analysts speculate that there is a current lack of investor demand for the precious metals space, and that a breakout above current resistance levels is needed before investors rush back into the space. For many analysts an important resistance level gold needs to break is around $1,360 an ounce.

Presented by www.kitco.com. Watch more here.

Related: Mark Cuban Says Gold and Bitcoin Are Equally Useless

More from Video

Inside Carnival's Mind Blowing New Horizon Cruise Ship (Video)

Inside Carnival's Mind Blowing New Horizon Cruise Ship (Video)

Jim Cramer: The 10-Year Yield Could Go to 2.75%

Jim Cramer: The 10-Year Yield Could Go to 2.75%

Elon Musk's Twitter Tirade Is the Dumbest Thing on Wall Street

Elon Musk's Twitter Tirade Is the Dumbest Thing on Wall Street

Replay: Jim Cramer on the Markets, 10-Year Yield, Oil Prices and Foot Locker

Replay: Jim Cramer on the Markets, 10-Year Yield, Oil Prices and Foot Locker

Video: You Could Live in a Ritz-Carlton or St. Regis Home

Video: You Could Live in a Ritz-Carlton or St. Regis Home