NEW YORK (MainStreet) — A large percentage of Americans are still shying away from investing in the stock market whether it is in a retirement portfolio or brokerage account.

Only 52% of Americans say they own stocks, according Gallup's annual economy and finance survey that was conducted in April 2013. Even fewer Americans, only 45% told the Pew Research Center in March 2013 that they owned individual stocks or stocks in a mutual fund in a retirement or brokerage account.

After the financial crisis, the number of stock holders dipped and many consumers refrained from getting back in the market, fearful that the huge losses they sustained previously might return.

Stock ownership varies widely by age and socioeconomic status and rises as the education and salary levels increase, according to Gallup.

In April 2008, more people overall owned stocks at 62%. The largest decline has been seen among 30 to 49-year-olds where only 58% owned stocks in 2013 compared to 72% in 2008.

The poll found that 77% of college graduates reported being invested in stocks compared to less than half of non-graduates and 80% of people with incomes of $75,000 or more compared with 55% of people with incomes of $30,000 to $75,000 who owned stocks.

Returns to the stock market continue to outperform investments in bonds or money market securities over long periods of time, said Robert Johnson, a professor of finance at the Heider College of Business of Creighton University Omaha, Neb.

Investors who allocated their funds in small company stocks yielded an average of 16.9% compared to 6.3% for long-term corporate bonds from 1926 to 2013, according to data published by Ibbotson Associates. Large company stocks produced a yield of 12.1% compared to U.S. Treasury Bills which generated an average of 3.5% in the same period.

The power of compounding demonstrates how far just one dollar invested in each asset class at the end of 1925 by the end of 2013 would have grown to $26,641.17 for small company stocks, $109.14 for long-term government bonds, but a mere $20.58 for U.S. Treasury Bills.

People continue to buy bonds even though the average annual return on them is much lower than stocks because bond returns are much more stable than stock returns, Johnson said.

"Over the long term, investing in stocks provides investors with higher returns, allowing them to eat better than bondholders," he said. "Investors take on risk in exchange for a return. Typically, the more risk or volatility that investors accept, the more they may expect in return."

The aftermath of the financial crisis is still fresh in many investors' minds, Johnson said. During the height of the crisis, many investors became "overcome with fear and capitulated," moving their assets from stocks to bonds and money market securities, he said.

"They have now watched the market rebound and the large indices have doubled in value," he said. "They fear that they could see a repeat of history if they put their money in equities now."

The best method to increase your retirement portfolio and to build wealth is to have an investment plan and stick to it. For most investors, dollar-cost averaging into equity mutual funds through either actively managed funds or index funds is a good strategy because it does not depend "upon the vagaries of the market," Johnson said.

"Those who pursued a dollar cost averaging strategy from the financial crisis through today have been rewarded handsomely," he said. "Remember Warren Buffett's advice: 'be fearful when others are greedy and greedy when others are fearful.' Unfortunately, most individual investors follow a herd mentality and are greedy when others are greedy and fearful when others are fearful."

Stocks have a long track record of producing higher returns than fixed income investments such as bonds, which is critical now that people are living longer, said Wayne Connors, managing partner of RetirementInvestor.com, a Glastonbury, Conn. website for DIY investors.

"They need to invest 'through' retirement to make sure their nest egg continues to have a growth component in order to provide an increasing income over their lifetime," he said. Behavioral finance dictates that many investors simply do not rebound from losing money.

People who suffered losses in the market feel the pain of losing money "three times as much as they sense the joy of gaining it and they have a tendency to not invest after they lose money," said Linda Duessel, senior equity strategist at Federated Investors, an asset management firm with over $350 million in customer assets, located in Pittsburgh, Penn.

Some investors are still living in the aftermath of the "Great Recession," which came just a decade after the collapse of the dotcom bubble. These huge setbacks in the market translate into a loss of "an entire generation of investors and many of them are still struggling in the underperforming labor market," she said.

Many of these investors are still underemployed or working part-time. A Gallup Poll last week showed that 57% of Americans believe the economy is getting worse.

"They can't fathom how the stock market is performing so well when the economy to them doesn't feel like it has recovered," Duessel said. "Since so many of them have had to make hardship withdrawals from their retirement plans, investing in stocks after the market has been so cruel to them is unthinkable."

Avoiding the stock market means Millennials and Gen Xers will see lackluster returns. While younger Americans are a lot more risk averse than previous generations, it is easy to see why, said Charles Sizemore, a CFA based in Dallas, Texas, who manages four investment portfolios on Covestor, an online marketplace for investing.

"If you were born after 1990, you're not old enough to know what a secular bull market looks like," he said. "Your enduring memories of the market are of watching your parents agonize over their losses in 2000-2002 and again in 2007-2008.You've also seen the destruction to a family's finances that a housing crash can cause."

The fear of seeing those outcomes being repeated in the market means investors will lose out on having enough money in their retirement portfolio.

"While their conservatism is understandable, it's also tragic because they are dooming themselves to a lifetime of disappointing returns," Sizemore said. "This has broader implications 30-50 years from now when they will most likely lack the funds to retire comfortably due to their lack of principle growth."

- Written by Ellen Chang for MainStreet