NEW YORK (MainStreet) — Withdrawing money from a retirement account is the quintessential financial no-no, but plenty of consumers are guilty of the mistake.

View Today's Highest Savings Account Rates

According to Fidelity, Millennials borrowed an average of 37% of their retirement balance to help finance a home purchase.

Savers who have time on their side are set to benefit from compounding interest in retirement accounts. Experts have traditionally advised saving for retirement in one’s early 20s, but withdrawing money early from the account interrupts this trajectory.

“The number of investors borrowing from their 401(k) has trended upwards in recent years, with more than two million investors now having an outstanding loan,” said Doug Fisher, senior vice president of thought leadership and policy development at Fidelity Investments. “Fidelity’s top concern is that within five years of taking a loan, 40% of borrowers decrease their savings rate, and more than a third of those stop saving altogether.

To illustrate the impacts, Fidelity ran the numbers for a hypothetical 25-year old making $50,000 per year. Upon initially saving 10% consistently for ten years, the investor would have $2,650 per month in retirement income, compared to just $1,960 if he had contributed 0% for 10 years.

“Reducing your savings rate today could significantly reduce your account balance upon reaching retirement and therefore your monthly income in retirement,” Fisher added.

Millennials are in a tough spot, especially when it comes to employment, wage growth and burdensome student loan debt. The generation has been postponing home purchases, compared to their parents and grandparents.

This delay is causing headwinds for the housing market, which thrives on increased demand.

Borrowing money from a retirement account before age 59.5 is inherently problematic, given the 10% penalty.

Given the today’s low interest rate environment, which won’t be here forever, it might make financial sense, especially if you’re struggling to put together enough money for a 20% down payment on a home.

Freddie Mac said the average rate on a 30-year fixed mortgage stands at 4.19%, compared to 4.22% at this time last year. The 10-year Treasury bond, which is the benchmark rate that helps set mortgage rates, currently yields 2.36%, compared to 3% at the start of the year.

Interest rates are expected to rise, given the end of the Federal Reserve’s quantitative easing program this month, which has kept interest rates low. Also, analysts anticipate the central bank will raise the short-term interest rates sometime in 2015, which will increase borrowing costs for banks and eventually consumers.

“It’s a good time to lock in a low interest rate,” said Bill Peattie, founder of Peattie Capital Management. “A $300,000 home will generally require a $60,000 down payment, so even if you borrow $10,000 from your 401(k) the penalty only amounts to $1,000.”

View Today’s Highest CD Rates

He says you should be able to make up that money, given the savings from low mortgage rates, if you plan to stay in the home long-term.

On the flip side, if you can’t come up with the down payment without the aid of raiding a retirement account, perhaps you’re not ready to purchase a home yet.

- Written by Scott Gamm for MainStreet. Gamm is author of MORE MONEY, PLEASE.