While 15 million Americans are without jobs, and perhaps ready to join another 2 million Americans in foreclosure, don't think they're the only ones with worries.

According to a just-released survey of millionaire households, having what would seem like a nice cushion of $3.5 million in investment assets -- outside a retirement plan and family home -- doesn't make a person immune from anxiety.

The annual Fidelity survey reveals that many millionaires, nearly half of those in the survey, actually don't "feel wealthy" and that they are taking steps to review their financial planning and even adjust their lifestyles.

The average age of those surveyed is 59, and many experienced the previous downturns from the 1970s oil crisis through the deep recession of the early 1980s with its double-digit unemployment rates. Still, 77% of those surveyed said the latest downturn is the "worst economic environment they have experienced."

The survey is done by an independent research firm, and participants don't know the results will be used by Fidelity. But Mike Durbin, president of Fidelity Institutional Wealth Services, notes that of the 4.4 million U. S. households with at least $1 million in assets outside their retirement plans, 37% have at least one account with Fidelity.

Millionaires Generally Optimistic

Perhaps the most surprising thing about the survey is the general level of optimism about the future and the stock market. Even though the survey was taken in late February, before the current market rally began, 39% said they felt confident about investing in the Dow Jones Industrial Average between 7000 and 7999.

And on a slightly longer time horizon, their optimism increases for 2010, bringing it to the highest level since Fidelity's "Millionaire Outlook" study began in 2006. In addition to anticipating a stock market rebound, there was increased optimism that business spending and consumer spending would rebound.

The survey revealed one huge cloud on the horizon for millionaires: the likelihood of higher taxes. Those expecting higher capital gains taxes was at 72%, while 67% anticipate a higher dividend tax rate, and 62% predict higher federal income tax rates.

They indicate they are taking action now, including accelerating plans to harvest remaining capital gains at the current low rates.

Changing Lifestyles

If you ever wondered why retailer Neiman Marcus is suffering, while Wal-Mart ( WMT) sales keep rising, it is explained in this survey. Although most Americans would feel wealthy beyond worry at the multi-millionaire level, those in the survey revealed they are cutting back their spending and planning to save more.

That could be because those surveyed revealed an average 19% drop in household income, and a similar drop in the value of their investments, along with a 28% decline in their real estate holdings.

Even so, nearly two-thirds in the survey picked the United States as the region with the highest potential for investment returns in the next year and five years, while China came in second.

Remember when you dreamed of making a million? (This presumes you are at least 50 years old, and were astounded at the young tech billionaires that emerged around the millennium.) Well, clearly a million dollars isn't what it was in the old days. It's not even enough to buy you weekly visits to a shrink who could help advise you on your anxiety over your shrinking assets.

Terry Savage is an expert on personal finance and also appears as a commentator on national television on issues related to investing and the financial markets. Savage's personal finance column in the Chicago Sun-Times is nationally syndicated. She was the first woman trader on the Chicago Board Options Exchange and is a registered investment adviser for stocks and futures. Savage currently serves as a director of the Chicago Mercantile Exchange Corp.

More from Personal Finance

How to Calculate Your Net Worth and Pin Down Your Financial Health

How to Calculate Your Net Worth and Pin Down Your Financial Health

The Best States for Millennials' Money and Health

The Best States for Millennials' Money and Health

U.S. Banks Urged to Make Small Loans In Competition With Payday Lenders

U.S. Banks Urged to Make Small Loans In Competition With Payday Lenders

How to Void a Check

How to Void a Check

Rent the Runway co-founder Chats with Cramer on Equal Pay and a Good Pair Jeans

Rent the Runway co-founder Chats with Cramer on Equal Pay and a Good Pair Jeans