U.S. investors have a case of the jitters these days, as the stock market has fallen more than 9% since the the start of the year and as U.S. economic growth, as measured by gross domestic product, fell to 0.7% last quarter.

Some of the savviest minds on Wall Street are quick to point out investors should not panic, and that's a fair point. Historically, investors screw up most when they pull money out of the market when prices are in full retreat, and thus miss out when stock markets soar skyward again, as they invariably do.

Below the surface, through, anxiety can be the default vibe for millions of Americans in a soft economy and a sour stock market. That angst is only enflamed if the investor suspects he isn't getting enough attention from his financial advisor, in a market period when the person needs his advisor most.

Now settle down, America's financial advisory community. Nobody is saying all, or even a significant number of advisors, aren't hustling to protect their clients' money during a tough economic period. But there are enough reports of advisors who don't adequately serve their clients, and who should be shown the door after periodic, even regular, instances of financial advisory malpractice.

Don't take it from us -- take it from your fellow financial advisory professionals.

"We hear story after story from women about their experience with financial advisors," says Carla Dearing, founder and chief executive officer of SUM180, a Louisville, Ky.-based online financial planning service designed specifically for women. "Common complaints are that they're expensive and condescending. But their concerns go beyond these. While they appreciate the help, they don't know if they can trust it. One woman told us, 'My husband's golfing buddy is our advisor, and that seems to be his only qualification."

To help women determine whether they have the right advisor, Dearing and her team asked the financial advisors servicing SUM180 clients for tips on how to determine whether they have the planner that's right for them -- or whether it's time to find a new advisor.

"Their advice goes a long way to ensuring a productive relationship with an advisor," adds Dearing.

Based on SUM180's research, and comments on tips from other industry experts contacted by TheStreet.com, here are five huge "red flags" indicating you (female or not) need to make a change from your current financial advisor:

1) Your advisor doesn't listen, or even worse, talks at you - Your financial planner may not be listening to you, SUM180 reports. "Clients -- whether individually or in a couple -- need to know they've been heard," Dearing said. The wrong advisor talks to hear himself or herself talk, the company states. The right advisor is constantly checking in to be sure the information being shared is understood.

2) Your advisor makes money on his specific investment recommendations - While registered investment advisors are fiduciaries, asking the simple question of "How do you get paid?" can shed light on where your advisor's loyalties reside. That's the take from Mike Jurs, director at Financial Engines, a San Francisco-based independent financial advisory service. "Don't worry -- you're not being rude," Jurs says. "It's almost expected. After all, it's your money and you should be sure that your advisor has your best interests at heart." Specifically, if your advisor earns a commission when he makes a transaction on your behalf, his priorities may be elsewhere, Jurs says. "And, if your advisor also creates the investments they'd like to put you into, their motivation to sell a particular investment could override your desire for long-term growth," he adds. "Take the time to understand how they earn money to minimize potential conflicts of interest and set a course for achieving your retirement goals."

3) Your phone calls and emails aren't being returned - The major sign that your financial advisor has checked out on managing your account is not hearing back on a call or note, says Pedro M. Silva, a financial advisor at Provo Financial Services, in Shrewsbury, Ma. "Depending on the size of the account and how the advisor has scaled his or her practice, clients might not be contacted as often as they would like, but a return call or email should be a basic courtesy in any industry," Silva says. "That said, I don't believe there is a need to 'confront' someone, as if it were a duel. I'm just saying if you're not getting the attention or performance you wished for, it's a good place to start the conversation."


4) Not keeping you in the loop - Susan Fulton, founder of FBB Capital Partners, a fee-only wealth management firm in Bethesda, Md., says an advisor who doesn't show you the performance on your investments net of fees and expenses needs to go. Same goes for one who is not readily available when it is convenient for you and who does not provide quarterly reports on the composition of your investments. "First and foremost, your financial advisor should be someone that you can trust," Fulton explains. "They should have a fiduciary responsibility in regard to you and your investments." If you do move on to another advisor, make sure to share with them what made you leave your last advisor, so they know the type of attention and reporting, you expect, she adds. "Make them live up to your standard."

5) Your advisor isn't following his own advice - Matt Ahrens, an associate financial advisor with Integrity-Advisory Group, in Overland Park, Kan., says he wholesaled for an insurance company for four years, before he transitioned to a financial advisory role. There, he saw firsthand advisors that "got it" and advisors that didn't. "Unfortunately the sign that most clients use is negative performance, but as all advisors know, performance is never within our control," Ahrens explains. "One red flag that is especially overlooked is whether advisors are following their own suggestions. Most advisors provide some sort of commentary to their clients, and if they say 2016 looks like a great year for municipal bond funds, but they don't adjust their allocation to include municipal bonds, then they're just blowing smoke."

When confronting an advisor, keep it simple but be honest, Ahrens adds. "If an advisor is hearing from all of their clients similar feedback, then it should be a wake-up call that something needs to change," he says.

One last word of caution, Ahrens notes.

"The last thing you want to do with your retirement savings is make an emotional decision so give your advisor a chance to make things right, if that's possible," he says. "But if you find yourself changing advisors every three or four years, then maybe it's not the advisor that needs to change."

More from Personal Finance

This Is How Hot It's Going to Get Where You Live

This Is How Hot It's Going to Get Where You Live

What Is a Timeshare and Is It a Financial Risk in 2019?

What Is a Timeshare and Is It a Financial Risk in 2019?

Retirement Advice - It's Actually Better to Work Longer Than To Save More

Retirement Advice - It's Actually Better to Work Longer Than To Save More

The Clock Is Ticking: Small Business Tax Strategies to Do This Year

The Clock Is Ticking: Small Business Tax Strategies to Do This Year

These Places Have the Highest Violent Crime Rates in the U.S.

These Places Have the Highest Violent Crime Rates in the U.S.