NEW YORK ( TheStreet) -- Mortgage rates are down this week, but the longer-term trend of rising rates over the past eight weeks has taken its toll, especially on refinancing rates.

Consumers rushed to refinance into lower interest mortgages in May and June, especially those consumers with sterling credit ratings -- 720 and above. Those mortgage shoppers could qualify for those under 4%, 30-year-fixed rate mortgages available last year and in the first few months of this year, but now 30-year fixed-rate mortgages are back up to 4.5%, down slightly from past weeks but trending up slowly nonetheless.

Big banks have already reacted to higher mortgage rates and lower refinancing volumes by shuttering mortgage financing units, or at least laying off hundreds of staffers. Wells Fargo ( WFC - Get Report) and Citigroup ( C - Get Report) have cut staff in their mortgage units after seeing weak demand for new mortgages, and especially for new refinanced mortgages.

Despite popular opinion, large banking executives aren't stupid. They see mortgage applications wane, and they see the handwriting on the wall.

According to the Mortgage Bankers Association's weekly mortgage application survey, mortgage applications were down 2.6% last week, and refinancing applications were down 4% -- signs that mortgage consumers have gotten wind of higher mortgages rates and are pulling back on new mortgage decisions.

The MBA says that U.S. mortgage refinancings are at their lowest levels since June 2011 and now represent only 63% of all total mortgage applications. The MBA's June Builder Applications Survey shows U.S. mortgage applications slid by 15% from May to June.

The association also points out that the average contract rate for 30-year-fixed rate mortgages jumped to 4.68% from 4.38% in the past 11 days, a number that would drive more mortgage consumers out of the market -- at least until rates recede to relatively reasonable levels.

With consumers less confident about the overall economy -- a Gallup survey out this week says that 52% of Americans believe the economy is getting worse -- mortgage rates could fall, as banks and lenders want to attract reluctant mortgage customers back into the market. That could lead to a temporary burst in refinancing applications.

But banks aren't betting on that, at least over the long haul.

Layoffs in mortgage units portend a mindset among lenders that rates will go up and mortgage applications and refinancing deals will go down -- and in the process, taking a big money-saving tool with them from U.S. homeowners.