NEW YORK (MainStreet) — As if you needed anything else to worry about at tax time, did you know there's a tax on gifts? The gifts you receive are totally free and clear; if you are the receiver you can more or less rest easy. But the gifts you give are subject to one of the most complicated areas of the tax code. And while it's a tax that mostly applies to the well-to-do, the threshold for what constitutes the "well-to-do" might be lower than you think.

The Origins of the Gift Tax

"It dates back to old tax law and philosophy," says Marc J. Minker, the private client services national practice leader with CBIZ, Inc. in New York. "Very few people today remember when we had marginal tax brackets in the 60 to 90% range."  He notes that through a lot of political wrangling, the marginal tax brackets were trimmed back, leaving the IRS with a problem: It needed to find alternative sources of revenue for the Treasury.

That's where the gift tax and estate tax regimes came into play. "They're sort of unified," says Miner. "You either pay the tax while you're alive or after you've passed away." The purpose of the gift and estate tax is to capture money that's not current income but saved assets that spill down from one generation to the next. Initially, Minker notes, there was just an estate tax. However, people very quickly caught on that they could simply transfer their assets while they were still alive to avoid the death tax. "They effectively pauperized themselves to avoid the estate tax," Minker said. Congress got wise to this trick and created the gift tax in 1932.

How the Gift Tax Works...

First, don't worry about documenting the $50 you spent on your mother for her birthday. The gift tax generally applies to larger gifts. In fact, unless you're giving more than $14,000 per year as an individual or $28,000 per year as a married couple, you can stop reading this article right now -- the gift tax doesn't apply to you. Still, the gift tax is more pervasive than you might think at first glance.

Adam Libman, an accountant based on Monrovia, Calif., notes the test of whether or not the money you're giving is a gift: "When you give the money, do you expect that it's going to be paid back, or is it a true gift?" There are certain exemptions for education and medical spending that will not impact your gift exclusion of either $14,000 or $28,000 per gift recipient. This means you can freely give away $14,000 to ten of your kids with no repercussions. But basically any other time you give money or goods away and don't expect to be paid back with interest or compensated with a fair market price, you're giving a gift.

But even if you do exceed that $14,000 or $28,000 number, you might not be taxed on the gift at all. That's because there's a lifetime exemption of approximately $5.43 million per person for giving away gifts. While you won't have to pay taxes on a gift of $50,000 as such, you do have to document the gift for the purposes of your lifetime exemption. "The IRS is getting much more sophisticated with their internal processing," says Libman. "They're going to see that money, and from the point of view of the IRS, everything is taxable until proven otherwise. So if they see that money moving around without a return, they're going to think it's untaxed income or even money laundering."

When a generous giver exceeds the $14,000 per person gift exclusion, he has to file Form 709 to the IRS to account for his lifetime tally. For example, on a $50,000 gift, a person would have to file Form 709 for $36,000. Of course, to reiterate: the receiver of a largesse is mostly in the clear, as long as the generous benefactor pays the taxes he owes on gifts. That said, the recipient may be on the hook for income tax implications if the gift generates interest, investment dividends or rental income in the case of property.  

As for the gifter, if his lifetime tally exceeds the $5.43 million exclusion, he's liable for a 41% beginning tax rate on the first dollar above the gift tax exclusion -- a rate that rises up to 45% depending on the size of the gift.  

Finally, in a community property state, you and your spouse have to have some kind of agreement on whether or not your giveaway constitutes a gift. Libman points out that's a lesson that Donald Sterling, former owner of the L.A. Clippers, learned the hard way when his affair was uncovered. And while you might not have to worry about gifts in the context of an affair, you do have to worry about whether or not that $20,000 you just gave your son as a down payment for a house is a gift or a loan.

Who Needs to Worry About the Gift Tax?...

As stated above, you only really have to pay a gift tax when you start giving away lots of money over the course of a lifetime. However, for the reasons stated by Libman, it's important that you document gifts, even if your gifts aren't going to clear the floor for taxable gifting.

Still, Minker notes that you might be closer to giving away $5.4 million than you might immediately think. He notes that, particularly for people in areas with a higher cost of living, like New York, Southern California and Silicon Valley, "little" things like the house you live in, a vacation home, some art on the walls and retirement accounts, quickly add up to $5.4 million. So while you might not part of the 1%, you might be part of what is referred to as the "mass affluent." That is to say, people with moderately above average income who have made some investments and own some real estate.

What's more, states with a high cost of living and a high percentage of mass affluent folks tend to have stricter laws than the federal government with regard to gifting, New York being a prime example. If you think that you might be brushing up against that floor, at the federal or state level, in life or in death, it's time to sit down with your accountant and talk strategy.