NEW YORK (MainStreet) — Can homeowner's associations (HOA) tell residents they cannot use, grow or distribute marijuana - even if the state says they can? The issue is gaining controvery in Colorado and other states where pot is legal.

"Homeowners associations can obviously enact restrictions banning illegal marijuana use or any other criminal conduct," said Bob Gaglione of Gaglione Law Group, San Diego. "But in jurisdictions where medical or recreational marijuana is legal, the answer is less clear."

Read More: An Overlooked Foreclosure Factor - HOA Fees

Gaglione said that in California, where medical marijuana is authorized for treatment of certain health conditions, HOAs can still limit pot smoking by residents.

"That's especially true, he said, "if the odors are not confined and disturb the quiet enjoyment of the use of property by neighbors, thereby creating a nuisance."

"But HOAs must be mindful of both the need to make reasonable accommodations for qualified individuals and the right to regulate the community for the overall good of the citizens," he added.

The reaction to the newly development of wide-spread legalized marijuana is sparking controversy among residents within HOAs across the country. There are 23 states where marijuana is legal to some degree. Several other states are placing it on the ballot. Still others, like Pennsylvania, have politicians who are trying to enact legislation.

According to an AP report, a Brighton, Colo. resident started growing hemp on a 75-by-100-foot plot in his yard. But his suburban Denver HOA was not having any of it and prohibited from doing so.

Activists offered to pay the fines, but the erstwhile hemp farmer declined. Instead, he sold the plants.

There have been similar incidents reported in other areas of the country. As residents complain about noticeable marijuana use, they look to their HOAs for a remedy.

Some marijuana proponents thinks people's attitudes will change. The conflicts in HOAs will disappear.

"I think [opposition] is misguided," said Jack Cole, founder and president of Law Enforcement Against Prohibition (LEAP). "I think the mindset will change. In Colorado, they are finding crime is down both property crime and violent crime. They will see that this is not in their best interest to do this."

A pro marijuana blog called Weedblog advises its readers, "Homeowner's associations can ban cultivation, but can't technically ban consumption. However, if a neighbor smells marijuana coming from your home, the homeowners association can take action. So while consuming on your own property can't technically be banned, all it takes is one complaint without proof to obtain the same effect as marijuana consumption being prohibited. This goes for legal states, medical or recreational."

Johnny Green, a prominent cannabis activist and blogger from Oregon, is also against HOA restrictions. "It's not right, and hopefully there will be challenge after challenge until all policies are updated to be sensible."

--Written by Michael P. Tremoglie for MainStreet