By Christina Cheddar Berk, CNBC News Editor

NEW YORK ( CNBC) -- Fears that retirement will lead to a drop in one's standard of living due to insufficient savings isn't a concern restricted to the middle class, according to the findings of recent survey.

Four in ten affluent Americans say their biggest fear about retirement is they "will do all the right things today and it still won't be enough for tomorrow" and 9% say they fear they have under saved and won't recover, according to a poll of 801 affluent Americans conducted by Harris Interactive for Wells Fargo Retirement.

"I think we just assumed they would be in a better place for retirement," said Karen Wimbish, director of Retail Retirement at Wells Fargo.
More from CNBC
Have You Committed Coupon Fraud Without Even Knowing It?
Businesses You Can Start for Under $5,000
What They're Googling Now

"Even among those considered 'well off,' many seem to fear a sharp drop in their post-retirement standard of living due to insufficient retirement savings," she said.

Other findings of the survey:

About a quarter of affluent Americans say they are not confident they will have saved enough for retirement, and this is especially true for Americans with assets between $100,000 and $250,000 (33%) and women (31%).

Among affluent women -- who are the lead drivers of America's consumer economy -- two in five, or about 42%, say they need to significantly cut back spending today in order to save for retirement versus 34% for men who feel that way.

That may be because affluent men have saved more than women, on average. The men in the survey had saved a median of $400,000 for retirement, but for women the median savings was $250,000.

Men also feel they need more money to retire comfortably. Men put that number at a median of $750,000, while women say $500,000.

There also isn't much difference between affluent Americans and the middle class when it comes to questions about when to retire. About three-quarters of the non-retired affluent Americans said having a specific amount of money dictates when one should retire, not the age one reaches.

That was a view shared by middle-class Americans polled in Wells Fargo's annual retirement study, which was released last month. That survey said a quarter of middle-class Americans, which was defined as those earning between $25,000 and $99,000 annually, expect to work until at least age 80.

Even among the affluent, 80 is the new 65. Nearly one in five with assets between $100,000 and $250,000 feel they will need to work until "at least age 80." Among all the affluent, 12% said they will work until 80 years of age.

In the survey, Americans had to have investable assets of $100,000, excluding real estate and other property, to be considered affluent. The group surveyed was split evenly between Americans with between $100,000 and $250,000 and those with more than $250,000.

--Written by Christina Cheddar Berk of CNBC.
CNBC is a world leader in business news, providing real-time financial market coverage and business information.