NEW YORK (TheStreet) -- The S&P 500 is down nearly 1% Wednesday, slightly off session lows. Biotech stocks continue to take the brunt of the selloff, with the iShares Nasdaq Biotech ETF (IBB - Get Report) down 3.6% on the day and 6.5% this week. The sector is up 240% since the beginning of 2012, causing some investors to be concerned about the valuation. 

Biotech stocks getting hit hard doesn't mean the sector will bring down the entire market, Josh Brown, CEO and co-founder of Ritholtz Wealth Management, said on CNBC's "Fast Money Halftime" show. 

The sector represents just 3% of the S&P 500, and is unlikely to cause any major correction. Last year, the sector fell 33% in the span of two months, while the overall market remained almost flat during that time. Investors should remain relatively calm, especially since there isn't a recession on the horizon, he said. 

There's definitely some "pause" in the biotech sector, which was rallying hard headed into this week, said Pete Najarian, co-founder of optionmonster.com and trademonster.com. Despite this week's losses, the sector is still up 13% in 2015. Investors are simply rotating out of sectors that have down well and putting that money to work in other sectors. 

Biotech shouldn't be viewed so unilaterally, argued Stephen Weiss, founder and managing partner of Short Hills Capital Partners LLC. It's the small-cap and mid-cap stocks that carry the most risk, while select large-cap companies, like Gilead Sciences (GILD - Get Report), have impressive growth and low valuations. 

Another worrisome sector on a valuation basis is technology. 

"I'm skeptical" of the prices, said Roger McNamee, co-founder of Elevation Partners. There simply seems to be too much excitement and not enough fear going into private market investments, like Uber, Snapchat and Pinterest, just to name a few.

While many of the companies are very exciting, the valuations seem to be a stretch, but only time will tell if that's the case, he added. A correction in the broader market could hurt private market investors and private equity firms as the valuations for many of these companies are pegged to the valuation of similar companies in the public market.

So does that leave retail investors at risk? Many big-name 401(k) fund companies including BlackRock (BLK - Get Report), T. Rowe Price Group (TROW - Get Report) and Fidelity (FNF - Get Report), among others, invest in these private companies. 

According to Mike Santoli, senior columnist at Yahoo! Finance, it's not something investors have to worry about right now. Considering the small size of many of these investments, even a collapse in the valuations would be unlikely to hurt the overall fund's performance by that much. However, there are some concerns when it comes to liquidity and calculating the net-asset value for the fund. 

In previous eras, the only way to get big funding was to go private, Brown said. That's not the case anymore for these public companies. It also gives fund managers a chance to buy into a company that's not in the public indices, meaning any gains they generate in these positions will purely boost their overall performance without boosting the performance of the traditional indices. 

In other words, it gives fund managers a chance to beat the index benchmarks by a larger degree than if the private companies were to go public. 

For their final trades, Brown is buying Valero Energy (VLO - Get Report), Weiss is looking for buying opportunities in biotech and Najarian likes chip stocks, particularly those that are suppliers to Apple (AAPL - Get Report).