Investing in bitcoin is as controversial as washing down a diet pill with a caffeinated energy drink. 

Speaking on the sidelines of the Vancouver Resource Investment Conference, well-known market pundit and bitcoin hater Peter Schiff said, "the number of people who bought into the market at $12,000 [a coin], $13,000, $14,000, is enormous...people had no idea what they were doing."

Schiff added that if bitcoin doesn't rebound from current levels, then "the brand is tarnished," and success stories about investors having made millions off the cryptocurrencies will change to people losing all their investments.

"Bitcoin is all based on faith, and when the confidence goes, all the value goes," Schiff said. "There are a lot of people that refuse to sell because they think it's going to a million, in the end it's all going to be wiped out."

On Thursday, bitcoin was last trading at $11,210 a coin. That's a far cry from late last year's all-time high over $19,000. 

More of What's Trending on TheStreet: