Sometimes hobbies can pay off, like the retired public works department painter in Massachusetts who amassed 1,200 paintings, sculptures and other works of art over three decades of scouring flea markets, auctions and galleries for finds.

James Pantages was once ashamed of his collecting habits but not so much anymore, with an appraiser estimating the whole collection may be worth $1 million, according to The Boston Globe.

So if you are an art lover, does it make financial sense to amass your own private collection? And what about buying classic cars, another popular item among collectors?

Well, it depends, say financial experts. While collecting art or cars may strike some as a complete waste of money, if you know what you are doing it may not be so crazy after all.

But following your passions is one thing, while investing based on the best returns available is another. You may not necessarily lose your shirt buying up paintings or classic sports cars, but it may not be the best investment, all things being equal.

"I am not suggesting that collectables are not a good idea," notes Richard Kagawa, a certified financial planner and president of Capital Resources & Insurance in Huntington, Beach, Calif. "Make sure that you have passion for whatever car, wine or art that you buy for investment because you may end up keeping it or not being able to sell it for what you want."

Diversifying your investment portfolio is an idea that is increasingly in vogue, with investment markets springing up around art and classic cars.

There are a number of market indices that track the investment value of art - Artprice's Global Index® claims a 30% increase over the past 20 years.

There's also a Dow Jones Investment-Car Index that tracks the returns on 50 classic cars.

The 1966 Aston Martin DB6 coupe is the successor to the car OO7 drove in "Goldfinger." The car, which can fetch anywhere from $68,000 to $95,000, has seen its value jump 63% over the past decade.

Not surprisingly, millionaires and billionaires often lead the way when it comes to investing in art.

Hollywood power broker David Geffen, who launched the careers of Tom Cruise, Joni Mitchell and many others and who co-founded DreamWorks, has amassed one of the world's most valuable art collections, pegged at more than $1 billion.

But it is a game that art lovers of even very modest means can pursue as well.

The late Herbert Vogel, a postal clerk, and his wife Dorothy spent decades amassing 5,000 works of art in their 450-square foot one-bedroom apartment in Manhattan. Vogel never made more than $23,000 at his day job.

The Vogels, who's love of art and an eye for talent, befriended up-and-coming artists in the 1960s before they were discovered and hit the big time. Their stash was so valuable that the couple could have become millionaires, but instead they opted to donate everything to the National Gallery and other venues in 50 different states.

Still, there are a number of factors you should consider before loading up on ultra-modern art works or classic cars from the 60s.

Eric Reich, president of Reich Asset Management in Marmora, N.J., says he understands the appeal, being a collector of classic cars himself.

"I don't advocate it as a retirement strategy, I will sometimes agree that as a means of DIVERSIFICATION it can make sense," Reich writes.

But there are other issues for investors to consider, with the illiquid nature of art, classic cars and other collector items.

While you can sell a stock in a matter of seconds, it could take a while to find a buyer willing to pay a good price for your cutting edge sculpture or your 1966 Aston Martin. In addition, like real estate, stocks and other assets, collectibles have their own cycles, with popularity waxing and waning for different styles, schools of art, or cars.

But the cycles are often longer than traditional investments and don't necessarily match up with your need to convert your prized painting or sports car into cash.

"This is a big potential problem when trying to sell as a good price when you want to," Reich says.

Then there are other pesky matters to consider, such as taxes, and fees for storage and insurance.

While long-term capital gains enjoy lower rates such as 15% or 25%, collectibles, no matter how long you have owned them, pay regular, short-term rates topping out at 28%, Reich notes.

And despite the stories of postal clerks and other everyday folks amassing art fortunes, George Gagliardi, founder of Coromandel Wealth Management, in Lexington, Mass., says investing in art and other collectibles is best left to the wealthy.

To play at this game, Gagliardi advises that you should have assets of well more than $10 million, an in-depth knowledge of art, cars or whatever your passion is, and the ability to ride out long periods waiting for the right deal to come along.

"Yes, a little more tangible than Bitcoin (which I avoid), but very high risk unless you really know what you are doing," Gagliardi says.