If you owe the IRS a whole lot of money and never address the debt, be prepared to hand over your passport.

Who knows? Uncle Same just might get around to taking it someday, if you ignore your bill.

In December 2015, Section 7345 of the tax code gave the Internal Revenue Service the ability to revoke your passport for nonpayment of taxes. Those outstanding taxes, interest and penalties have to add up to $51,000 or more during the last 10 years, a notice of lien has to be filed against you and a levy has to be issued, but it can happen.

Back in January, the IRS finally started taking action against individuals with "seriously delinquent tax debts" and warned that it would notify the State Department of taxpayers meeting that standard. The law requires the State Department to deny those taxpayers a passport application, renewal of their passport, or in some case, revoke their passport entirely.

That said, there are several loopholes to that particular law. If those delinquent taxpayers are in bankruptcy, victims of tax-related identity theft, declare hardship, are in a federally declared disaster area or have requested an installment payment plan from IRS, their passports are safe. If they're currently serving in a combat zone, the IRS will postpone notifying the State Department until they return.

In fact, to have your passport taken, you would have to completely reject installment payments, wave off an offer in compromise or settlement agreement with the IRS and reject a collection due process hearing to determine the amount of the levy owed. Then you'd have squander the 90 days the State Department gives you to sort out errors in your outstanding debt, pay the debt off entirely or enter an IRS installment plan.

"There really has to be willful blindness on the part of the taxpayer to get into this kind of situation," says Barry Weisman of Anchin, Bloch and Anchin. "Just truly ignoring or being in denial about an IRS assessment."

Yet passports aren't the only documents imperiled by debt. States have resorted to suspending or revoking driver's licenses to coax the payment of outstanding tax debt -- with Weisman noting that New York does so for debt of $10,000 or more. While only 36% of U.S. citizens hold passports, according to the State Department, a large portion of that subset tends to live outside the country and would run into serious issues if those passports were revoked.

"First, they would typically use their passports more often -- not only for travel but for administrative matters, such as rental contracts, in their countries of residence," says Nigel Green, founder and chief executive of U.K.-based finance firm deVere Group. "And second, since the worldwide rollout of the highly controversial Foreign Account Tax Compliance Act in 2014, tax returns have become more complex, onerous and burdensome for U.S. expats due to additional reporting requirements."

In Green's experience working with U.S. citizens who live abroad, 35% are likely to make a mistake on their tax return and/or file late due to both FATCA and Section 7345. However, Weisman notes that any outstanding debt would have to be assessed first. If there isn't an outstanding assessment, Section 7345 doesn't apply. If there's been an assessment, you, your attorney or your accountant can request a transcript of your account from the IRS just to clarify how much you owe.

When that transcript arrives, just keep in mind that your debt doesn't have to be strictly related to income taxes. If employment taxes have been assessed against you or you were audited years ago and lost track of the results of that audit, what you owe the IRS could easily balloon to $51,000. While that's certainly a problem for dual citizens or expats, taxpayers who haven't given their passport a workout in a while can run afoul of these issues as well.

"The IRS is targeting a relatively small group of taxpayers that has either forgotten about it or is living overseas and doesn't think it applies to them, or just ignored it and wished it away," Weisman says. "There are people who owed a lot of money on their tax return for improper withholding, no withholding or they just didn't pay estimated taxes."

DeVere's Green says the uncertainty surrounding Section 7345 shouldn't give expats a false sense of security. Their passports are far too valuable to gamble with, which is why they should seek assistance with their outstanding tax assessments.

"For U.S. citizens who are resident overseas, the IRS' latest weapon to collect taxes means it is more important than ever to stay on top of your taxes and file on time and correctly," he says.

This article is commentary by an independent contributor. At the time of publication, the author held no positions in the stocks mentioned.

More from Taxes

Who Knew an IRS Tech Glitch Could Get You More Time To Prep Your Tax Return

Who Knew an IRS Tech Glitch Could Get You More Time To Prep Your Tax Return

Tax Snafus Can Block Legal Residents' Path to Citizenship

Tax Snafus Can Block Legal Residents' Path to Citizenship

Tax Tip: Get Money Back on Those Pesky Kids

Tax Tip: Get Money Back on Those Pesky Kids

Tax Tip: How To Report Your Bitcoin Earnings

Tax Tip: How To Report Your Bitcoin Earnings

Tax Tip: Uncle Sam Will Be Looking for Your Cryptocurrency Trades

Tax Tip: Uncle Sam Will Be Looking for Your Cryptocurrency Trades