As the energy industry takes a hit from lower oil prices, big banks are feeling the pressure, analysts say.

"In terms of banks and energy, essentially, virtually every major bank involved in lending money to the energy industry is taking increases in their reserves and taking hits in loan losses in their first quarter," said analyst Dick Bove of Rafferty Capital Markets.

Wells Fargo & Co. (WFC)  was hit particularly hard. The company announced first-quarter earnings Thursday morning and while revenues increased, investors aren't exactly looking to buy more shares.

"Wells Fargo is being affected right now," Bove noted. The company had $204 million in net charge-offs for its oil and gas loan portfolio, up $87 million compared with the fourth quarter of 2015. Oil and gas accounted for 1.9% of total outstanding loans.

The company's report, was not as bad as expected, however, most likely due to an uptick in the residential real estate market.

"When investors just focus on energy, they don't see that the overall portfolio is still healthy," said analyst Marty Mosby of Vining Sparks. "The uptick in residential real estate is helping them reallocate funds. They're helping make the impact much more manageable."

Though analysts predict that oil and gas's downturn will affect the banking industry, the hit may not be as hard as traders feared in January and February, when crude oil sank to nearly $26 a barrel from a 2014 peak above $100.

Another plus is that energy loans are a mere fraction of the size of the $15 trillion mortgage industry, whose cascading defaults starting in 2007 led to the failure of investment bank Lehman Brothers and a global credit crisis.

"Residential real estate was a third of the banking industry's portfolio as a whole," Bove said. "Energy is one thirtieth."

Banks in oil-rich regions of the U.S., especially those with large fracking operations, which use hydraulic drilling to draw oil from shale formations, are likely to feel greater impacts since low prices have curbed production. Those, according to Bove, include Capital One  (COF) , which is headquartered in McLean, Va., and Comerica  (CMA) , which is headquartered in Dallas.

Mosby agreed, pointing out the risk to regional banks in areas like Western Pennsylvania, Texas and North and South Dakota.

This, though, doesn't exclude big banks: Mosby said Zions Bancorp (ZION) has one of the largest concentrations of investments in the energy industry.

"It is definitely giving investors for banks pause before they buy into it," Mosby said.

"Wells' oil/gas loan book, which is roughly 1.9% of total loans and deteriorated in the quarter -- although not as bad as many may have feared -- doesn't do the investment case any favors in this uncertain environment," said TheStreet's Jim Cramer, who holds the stock in his Action Alerts PLUS charitable trust.

Wells Fargo, which has a market cap of $247.75 billion, was trading at $48.99 midday Thursday, down 16% from its 52-week high of $58.77 on July 23.

Capital One, which has a market cap of $37.23 billion, was trading at $71.72 per share midday Thursday, down 22% from its 52-week high of $92.10 on July 22.

Zions, which has a market cap of $5.28 billion, was trading at $25.83 midday Thursday, down 21.7% from its 52-week high of $33.03 on June 21.

Comerica, which has a market cap of $7.01 billion, was trading at $40.06 midday Thursday, down 25% from its 52-week high of $53.45 on June 24.

EXCLUSIVE LOOK INSIDE: Bank of America and Wells Fargo are holdings in Jim Cramer's Action Alerts PLUS Charitable Trust Portfolio. Want to be alerted before he buys or sells the stocks? Learn more now.