While most small and mid-sized banks based in the U.S. were not directly affected by Britain's decision to leave the European Union, their share prices have taken a hit along with larger competitors.

Still, there could be an upside, analysts say, particularly for stocks like Pinnacle Financial  (PNFP - Get Report) ,  ServisFirst Bancshares  (SFBS - Get Report)  and Renasant  (RNST - Get Report) .

Since those banks typically trade at higher multiples to their financial performance, investors might view Friday's drop as a chance to buy the stocks at lower prices, according to analyst Joe Fenech of the Hovde Group. Another plus is that the firms are less exposed to oil prices, which have fallen 7% since Britain's vote and dragged down the shares of energy lenders.

"Since Friday you've seen the most asset-sensitive bank stocks are the ones that are underperforming," analyst Michael Rose of Raymond James said in an interview. "What it implies is that consensus expectations need to be revised downward."

Fenech agreed.

"It will likely be harder to pound the table on a positive thesis that had previously prominently featured the benefit of asset sensitivity," he wrote. "It may also be more difficult to point to higher/more stable oil prices as a catalyst for energy-exposed names, as there could be increased volatility in coming days/months."

Before the vote, oil prices had begun to rally, reaching about $50 a barrel from a low of about $26  in February amid a global supply glut. That's still more than 50% below the peak of above $107 in 2014.

According to Rose, Comerica  (CMA - Get Report)  and Texas Capital  (TCBI - Get Report)  were some of the hardest hit by the Brexit decision, thanks to their location in the Lone Star state and their exposure to the oil market.

"We upgraded the energy-exposed banks seven weeks ago, but those are also the most asset-sensitive," Rose said. "It'll be interesting to see what happens over the next few weeks."

Britain's decision to leave the European union, and the economic volatility it has already caused, makes it less likely that the Federal Reserve will deliver a much-anticipated interest-rate hike this year, Rose said. Raymond James predicts no interest-rate hikes for the rest of the year and only two in 2017.

That might prompt banks to lower their earnings outlooks for the remainder of the year, which could drag down share prices before second-quarter earnings, which companies will begin to report in July, Rose said.

So what should investors do? Rose suggested avoiding larger banks, even if "valuations are tempting."

Fenech advised being a bit more risk-averse in the short term. "Companies with demonstrated problems that are in need of capital might now require a longer look at the very least," he wrote.