There are a few strategists out there telling investors that the U.K.'s Lloyds Banking (LYG) - Get Report is a good stock to buy on weakness. Some even call it a contrarian play. But let me be contrarian to the contrarians and say that investors should really, really take a good look at this one before making any such commitment.

And it's not because its chief executive recently apologized to staff for allegations of an affair; really, that's between him and his wife. It shouldn't affect the bank or the stock price in any way. No, the reason I am saying investors should be careful on this stock is simply that Lloyds, perhaps more than the rest of U.K. banks, could find itself pretty shaken by the Brexit.

Those who look at the stock with enthusiasm wax lyrical about its potential for recovery. Indeed, shares have lost more than 20% year to date on the London Stock Exchange (where the bank's main listing is trading under the symbol LLOY), but are up some 1.8% since the Brexit vote's results came out on June 24.

The consensus of 28 analysts covering the stock, as quoted by the Financial Times, is that Lloyds will outperform the market. Their median price target is 67 pence, which means upside potential of 15% from Friday's closing price. (Monday is a bank holiday in the United Kingdom, so the London-listed shares won't trade.)

Enthusiasts say that Lloyds Bank's big exposure to the U.K. domestic market and its relatively low exposure to the rest of the European Union means it could potentially outperform other British banks such as Barclays (BCS) - Get Report , Royal Bank of Scotland (RBS) - Get Report or even HSBC (HSBC) - Get Report after the Brexit. But I would argue that it's just the opposite.

While Lloyds indeed wouldn't suffer as much as some other major U.K. banks because of the withdrawal of "passporting" rights (which give U.K. banks the right to do business in all other 27 E.U. countries), the fact that it's exposed to the U.K. housing market as the country's biggest mortgage lender isn't encouraging.

The latest murmurs from various E.U. capitals don't bode well for the Brexit negotiations, either. Take Germany's outspoken Vice Chancellor Sigmar Gabriel, who said the Brexit could send the E.U. "down the drain" if other states follow the U.K.'s example. What this means is that E.U. leaders will do all they can to ensure that the United Kingdom isn't better off outside of the union than in. It's way too early to speculate on what exactly that entails, but one thing's sure: the negotiations will likely be acrimonious and long.

The U.K. government's hopes that it could negotiate a broad agreement with the European Union before actually triggering the E.U. treaty's Article 50 covering exits from the union have been dashed again. One E.U. diplomat told Reuters that invoking Article 50 is like switching off the engine on an airplane -- it's best done when a landing strip is in sight.

But a source close to German Chancellor Angela Merkel told Reuters that the German leader stands by her decision not to hold any significant talks before Britain officially triggers Article 50. So, the United Kingdom either risks switching off the engines in mid-flight or rushes to find an airport without really being sure it's the right one for landing.

Things aren't looking so great inside the United Kingdom, either. On the political front, Prime Minister Theresa May is due to meet her "three Brexiteers" -- Foreign Secretary Boris Johnson, Trade Secretary Liam Fox and Secretary of State for Exiting the European Union David Davis -- on Wednesday to discuss an action plan. There are reports that the three men have daggers drawn, with egos clashing and senior civil servants warning of "huge frictional costs."

On the economic front, the Bank of England has done a lot -- slashing interest rates and restarting bond buying -- but can't keep throwing money into a huge pit of uncertainty. (Well, it can if it wants to, but to little effect.)

But uncertainty is already taking a toll on the British economy. U.K. house prices were in bubble territory but have begun to cool, and anything that could tip them over into a crash would be very bad news for Lloyds as the country's biggest mortgage lender.

It's true that the Bank of England can keep bailing out homeowners with low rates and money printing as long as investors trickle out of the U.K. housing market in an orderly way. But the BoE wouldn't be able to do so if a trickle turns into a flood. In that case, big exposure to the U.K. market would become a liability for Lloyds, not an asset.

Editor's Note: This article was originally published on Real Money at 8 a.m. on Aug. 29.

Employees of TheStreet are restricted from trading individual securities.