NEW YORK (MainStreet) — The saying goes, "a rising tide lifts all boats," but that's simply not always the case when it comes to investing. With the record-setting bull market now nearly six years old, some investors are still falling behind. While the S&P 500 rose over 11% last year — its third straight year of double-digit gains — a large group of young adults and women are missing what some market watchers are calling a secular bull market.

In a survey of 1,800 affluent investors conducted by Sullivan, a brand engagement firm, and Northstar Research Partners, 59% of Millennial investors and 36% of female investors had flat — or even lower — portfolio returns. It's a matter of appropriate risk-taking. Adults under 35, who entered the workforce just as the Great Recession slammed the economy, are generally unwilling to buy into what they consider to be "risky" investments. In fact, Millennials are almost twice as conservative today as they were in 2012. Meanwhile, women are 12% more likely to be conservative investors than men.

It's one of the main roadblocks preventing investors from achieving their retirement goals, and that fear of investment losses goes hand in hand with another emotional driver that can put investors in a lockdown mode: a lack of confidence.

The research found that investors are concerned about health care costs – whether due to rising expenses or a sudden, serious illness. When investors admitted a lack of confidence in their investment strategy, fully four in 10 (44%) cited health care costs as the leading reason — followed by the fear of a major market decline.

"While our study has found a revitalization in the relationship between investors and financial institutions, investors are hungry for more help from their advisors — beyond just advising on portfolio transactions," says Nancy Schulman, partner and executive director of strategy at Sullivan. "In a climate where the markets are up and trust is high, some investor segments have been left behind, and broad-based issues like health care costs need to be integrated into the conversation."

In addition to the portfolio paralysis that can prevent an investor from taking prudent action to grow their retirement savings, more than half of investors aren't entirely forthcoming when discussing their investments, failing to disclose an average of 30% of their assets to their financial advisor.

“In conducting this research over the past five years, we’ve tracked tremendous change among the affluent investor community," says Tricia Benn, managing director of Northstar. "Key investor concerns uncovered in this latest study show opportunities for deeper engagement between financial institutions and today's affluent investors, especially as it relates to health care costs, employer-sponsored retirement plans and the role risk plays in portfolio performance.”

And finally, one more roadblock to investor success was revealed: The mystery of performance-crippling fees. More than one quarter (27%) of the affluent investors surveyed didn't know how much they were paying their advisors — half of those (14%) being unaware they paid any fees at all.

Hal M. Bundrick is a Certified Financial Planner and contributor to MainStreet. Follow him on Twitter: @HalMBundrick