The woes of the U.K.'s financial heavyweights have stolen all the limelight since the Brexit vote was announced on Friday. However, London's small collection of so-called "challenger" banks could fare worse in a U.K. retreat from the European Union.  

In the aftermath of the 2008/2009 financial crisis, Britain's biggest banks quickly became political whipping boys after multi-billion pound state bailouts. "Too big to fail," they sailed close to the regulatory wind and could do that in part  because consumers had no viable alternatives, or so the charge went.

During the intense period of "banker-bashing" that followed the crisis, lawmakers sought to encourage new entrants into the banking market.

Vernon Hill, the founder of Commerce Bancorp, established Metro Bank (USMT) in London in 2010 and today, the group has 41 branches and GBP7 billion ($9 billion) of assets. 

Virgin Money is another challenger. It is backed by British billionaire Richard Branson and listed on the London Stock Exchange back in 2014.

Other challenger banks include ShawbrookTSB Bank, which was taken over by Spain's Sabadell  last year, Svenska Handelsbanken (SVNLY) , Aldermore and OneSavings Bank (OSVBF)

OneSavings and Virgin Money were among the biggest fallers in the sector on  Friday and Monday, with both of them losing close to 50% of their value. That compares with the 30% average sustained by their larger peers Lloyds Banking (LYG) - Get Report , Barclays (BCS) - Get Report and Royal Bank of Scotland. 

On Tuesday a new shock came as Shawbrook admitted it had inadvertently broken its own lending criteria in its asset finance business and would set aside GBP9 million against the ensuring bad loans.

Earlier, Branson claimed in a TV interview on Tuesday that his company had already cancelled one deal because of the vote to leave the EU. He did not give details  but said that it would have been worth 3,000 jobs to the U.K. economy.

Branson, a pro-European Union campaigner, has called repeatedly for the referendum to be re-run.

If the fallout from Brexit for the British economy is as bad as many seem to be predicting, with interest rates going to zero and the economy stalling, then net-interest margins will suffer across the financial world. This is particularly bad for challenger banks given the smaller scale of their businesses and the fact that they lack international and product diversification. 

Larger competitors like HSBC are internationally diversified, while Lloyds Banking also has large commercial banking and insurance operations as well as consumer banking. Barclays has a global trading and investment banking business that still provides around GBP8 billion ($10.1 billion) in revenue to the group each year. 

But the U.K. challenger banks do not have corporate, trading or investment banking operations that they can rely on to make up for the interest-margin shortfall. Nor do they all necessarily have the scale to weather downturns. 

Some, such as OneSavings, Aldermore and Shawbrook, have buy-to-let mortgage finance businesses. But it remains to be seen whether margins earned from landlords will be enough to save the challengers from a net-interest cliff in the consumer and SME businesses. 

On Tuesday Shawbrook closed down almost 15% at 140 pence. It held an IPO last year at 290 pence.  Virgin Money and Metro were up almost 9% and more than 7% respectively as banking stocks recovered in a rising market. Aldermore closed up 3.9%.