NEW YORK (MainStreet) — First there was the concern that you'd never pay off your student loan or, if you did, it would take decades. It has become the not-so-new normal.

It's also more than a life-style cramp. Now the fear is that student loan debt can push people from the middle class — or deny them entry — while their wealthy, student debt-free peers have nowhere to go but up.

Those are among the unintended consequences of borrowing money to go to college — the loan balance for the average graduate is closing in on $30,000. Paying off principal and interest on loans prevents borrowers from making other investments, not just in cars and home mortgages, but stocks, bonds and financial instruments in general—including those found in a retirement account.

William Elliott, director of the Assets and Education Initiative at the University of Kansas said that that in 2009 median net worth for households with no student debt was three times what is was for households with no student loans. Few expect that trend to be reversed.

"Education is often imagined as a driver of equality," he said in the preamble to an address at the University last November. "But higher education operates more like a commodity purchased by individual consumers. As a result, cost burdens fall on students and families who frequently rely on debt to pay for college. Rather than equalize educational and economic outcomes throughout students' lives, college largely reinforced patterns of privilege."

Some of the origins of this problem aren't especially well known. For many, it starts in high school, where grants and scholarships, which don't have to be repaid, are less available to those who need them the most. Experts have noticed a trend where aid is becoming merit-based rather than need-based. This type of aid goes to higher-achieving students who tend to come from higher income families.

Donald Heller, dean of Michigan State University College of Education told the Associated Press last week that, "Because there's a strong correlation in this country between things like SAT scores or ACT scores and wealth or income, the (grant) money ends up going disproportionately to students from wealthier families" who tend to perform better on those tests.

Those factors, along with stagnating family incomes and declining savings, have made student loans a much bigger part of funding higher education, said Elliott of the University of Kansas.

If the notion that a wealth gap is not new, the idea that higher education may be increasing it rather than making it smaller is novel—and disturbing. Michael Norton of Harvard Business School ran a 2011 survey that found that people tend to think wealth is more equally distributed than it really is and the gap is actually getting wider. The Obama administration has noticed and proposed changes last August to federal student aid programs that would link federal dollars to a college rating system that rewards schools graduating a higher number of low income students.

Whether the Obama plan gets through Congress is another matter. But as Norton points out, "Both political parties are now saying that perhaps inequality has gotten to the point where it's not fair when people don't have a chance to rise, and we need to do something about it."

College degrees can pay off. College graduates ages 25 to 32 working full-time earn $45,500, about $17,500 more than their peers with just a high school diploma, according to a 2012 Pew Research Center analysis of census data. But by the same token, cost is influencing graduation rates and feeding the worst-case scenario, where debt forces people from school before they get their degrees.

"We can't," Norton said, "let debt hinder a whole generation of people from beginning to accumulate wealth soon after graduating college."

--Written by John Sandman for MainStreet