The first few months of 2016 saw elevated levels of volatility in global stock markets amid falling oil prices and worries about a slowdown in China's economy. Some 40% of investors think that volatility is here to stay, according to a new survey released Thursday by Wells Fargo.

"Clients look at the portfolios and they look at the markets and they just see up 300 points one day and down 400 points the next," said Joe Ready, director of Wells Fargo Institutional Retirement and Trust. "I think it brings back nervousness and a concern with the economy and if we're going to go back to a recessionary period."

After all, 2016 saw the worst start to a new year for stocks in history and the volatility continued well into March.

Another four out of ten respondents in the survey, which was taken during first quarter, said the volatility in the markets is adding stress to their lives.

"It's clearly starting to wear on investors," Ready said, adding that the percentage of investors who felt confident that the stock market was a place to save and invest dropped to 36%, from 43% during fourth quarter.

Despite the steep swings in the markets, 81% of respondents stayed the course and continued their investment plan, instead of panicking and selling everything. Only 4% sold stocks following January's roller coaster ride across global markets. Some 11% actually used the price swings as a buying opportunity.

"I think people are looking at their 401(k)s, and although they'll see volatility, they realize it's for the long-term," he said.

While 401(k) balances may have been dinged so far this year, gas prices are still roughly $0.60 cheaper per gallon than they were last year, according to AAA.

Wells Fargo's survey said 37% of consumers are throwing the gas stimulus towards paying off debt and 31% are boosting their savings accounts.

"Clearly what's going on, whether it's within their 401(k) or their own personal household budgets, they're improving their financial wellness," Ready added.