NEW YORK (MainStreet) — If you've just bought a home and have a brand-new mortgage, the last thing on your mind is refinancing. After all, if you shopped carefully, you already have the best terms you could get. And odds are that mortgage rates will go up, not down, making a refi a non-starter anytime soon.

Still, the unexpected does happen. So it could make sense to begin building a fund for a future refinance, enough to cover the thousands in fees you might have to shell out for a better deal. Many homeowners find it pays to refinance even if they reduce the loan rate by only a fraction of a percentage point.

Obviously, that opportunity could present itself if mortgage rates were to dip. With the 30-year fixed-rate mortgage hovering at about 4.1%, a drop might not seem likely. But two years ago that rate was around 3.6%, and a half percentage-point drop could easily justify a refinance if you expect to stay in the home for five or six years, long enough for the monthly savings to offset refinancing fees. (Use the Mortgage Refinance Breakeven Calculator to see.) The mortgage market has repeatedly surprised experts who expected rates to rise faster sooner. So who's to say rates absolutely, positively, won't drop?

If you've built up some equity in the home, or have funds to pay down the loan balance during a refinance, you also might be able to escape mortgage insurance charges while switching to a new loan. That, on top of any rate cut, would allow you to break even on refinance costs even sooner.

Also, you might get a new loan with a lower rate by paying points, or upfront interest charges. If you've built up some cash to pay points, or now think you'll stay in the home longer than you'd thought before, a refi with points paid could save you money in the long run.

Even if average loan rates don't come down much, you might be able to get a cheaper loan if your credit score improves. A small rise that moves your score from one category to another could qualify you for a loan that would be 0.25% cheaper.

Another reason to refinance: you might want to switch from one type of loan to another.  A borrower who believes interest rates will stay low for a few more years might switch from a fixed-rate to an adjustable-rate mortgage. One-year ARMs start at an average of about 2.6% now, and three-year ARMs just over 3%. 

On the other hand, if you expect rates to rise, you might want to refinance to trade your current ARM for a fixed-rate loan.

Or, if your income has gone up and you can handle bigger payments, you might want to switch from a 30-year fixed loan at, say 4.1%, to a 15-year deal, averaging about 3.5%. The payment would be bigger because you'd pay more principal every month to retire the debt in 15 years instead of 30, but your interest costs would be much lower over the loan's life.

Because home prices have gone up substantially in some markets in recent years, a home you bought two or three years ago might be worth a good deal more than your mortgage balance.  With a cash-out refinancing, you could turn that equity into cash for other purposes.

In other words, you never really know what's going to happen, so it pays to have some cash on hand to seize an opportunity.  Closing costs range from 2% to 5% of a home's value, or $2,000 to $5,000 for every $100,000, according to Zillow.com

And if you don't refinance, your refi cash will boost your rainy-day fund. It never hurts to have a bit more in reserve.

By Jeff Brown for MainStreet