NEW YORK (MainStreet) — There are few minor annoyances more irritating than bank and credit card fees. Maybe your payment arrived late. Maybe you're overdrawn by a few dollars. Maybe you didn't meet the requirements for a "free" account this month. Or maybe you just don’t want to pay your annual credit card fee. So how do you get out of paying your fees to banks and credit card companies so you can spend more on yourself?

Step One: Call Up and Ask

"A lot of people don’t realize this," says Mike Sullivan, director of education for Take Charge America, "but if you call up customer service, a lot of times they're empowered to waive a fee for the first request."

What's more, a lot of times they're authorized to refund fees for the second request you make. They might even be allowed to give you yet another fee waiver, depending on how much money it’s for.

Sullivan points out the bind credit card companies and banks are in when it comes to such fees.

"They make a lot of money off of these," he says. "If someone makes a late payment every month and owes them $35 every month, that's good money for them."

However, it also costs a lot to get new customers. This means they'll often waive fees for customers in an attempt to get you not to jump ship to another credit card company or bank.

Before you call, however, Sullivan suggests you think about whether or not it's worth it.

"If it's $35, I'd call,” he says, “If it's only $5, I'd save that goodwill for more significant charges."

WATCH: More personal finance videos on MainStreet | More videos from Amelia Martyn-Hemphill

View Today’s Auto Loan Refi Rates

Step Two: Make the Case

Randy Padawer, a consumer advocate with LexingtonLaw, is also a proponent of calling up and getting fees back. He encourages customers to make the case for themselves. "You shouldn't just call up and say, 'I'm tired of paying my annual fee.'"

So what should you say?

Padawer uses a rewards card as an example. If the card is tied to a specific airline or hotel chain you're not flying on or staying at, go ahead and tell the card issuer that. You're not going to get out of the fee for good. The customer service agent might, however, be able to waive it for a year to encourage you to keep the card and, more to the point, start staying in the specified hotels and flying the particular airlines.

Another example is when you're a good customer with a lot of accounts with your bank. Your case might be as simple as "I’m a good customer who never overdraws my accounts." The bank isn't going to want you to move your accounts somewhere else, so many times they’ll be more than willing to refund some or all of the fees you’ve incurred.

Step Three: Be Nice, But Escalate

"I always find it better to be polite when talking to customer service," says Sullivan. Being polite has a pragmatic edge to it.

"[Representatives] are generally empowered to give you something," Sullivan said. "Making them want to give it to you can really help."

So what happens if you start off being nice and you don’t get what you want?

If the first agent can’t give you a satisfactory solution, escalate it to a supervisor.

"The supervisor is the only one I speak to sharply and talk about closing my accounts," he says. He's also quick to add that he doesn't talk about closing his accounts unless he's fully prepared to do so. Sullivan says he's never had to bring issues to yet another even higher supervisor, but you'll generally be able to talk to one during normal working hours.

What if that doesn't work? At that point, you can send a letter to the president of the bank or some kind of senior official. Still, he notes that this is only really appropriate when there has been a serious professional, moral or legal transgression.

"The president of the bank doesn't want to know you were charged $35," he said.

Step Four: Cancel or Downgrade Accounts

When all else fails, you can go ahead and cancel accounts, but there's also downgrading. What’s that?

"Most credit card companies have different levels of membership," says Padawer. Similarly, you might have one kind of bank account, but instead opt for a different, less expensive one. Padawer notes that a lot of the time, companies will provide the same benefits, but waive the fee, because that's the easiest possible solution.

If you're looking to waive fees, you need to be proactive, be polite and pick your battles. If you can maintain your composure and make a solid case for yourself, you'll start paying less to other people and having more for yourself.

-- Written for MainStreet by Nicholas Pell