Updated from 1:40 p.m.

The BlackBerry patch got thornier Wednesday for Research In Motion ( RIMM).

A judge denied RIM's motion to enforce a previously reached patent settlement, ruling that a valid agreement was never reached. RIM shares fell 5% after being halted for two-and-a-half hours.

The stock has dropped 30% this year as Virginia's NTP has pressed a case claiming the BlackBerry messenger infringes on its wireless patents. RIM agreed earlier this year to pay $450 million to settle, but the deal fell apart in June.

The ruling leaves the fate of the BlackBerry with the federal court system and the U.S. Patent and Trademark Office. The case is currently at the District Court level after an appeals court largely upheld a ruling that RIM had infringed on several of NTP's patent claims.

"This opens the whole can of worms all over again," says Ovum analyst Roger Entner. "The refusal of the judge to enforce the agreement puts the whole settlement in doubt and brings them back to square one."

Investors had been hoping for some sign that an end was near for the prolonged legal battle. But now NTP is in a much stronger position and wielding the upper hand in negotiations. RIM shares fell $3.30 to $61.62.

RIM said in a press release that it will appeal the decision. The Waterloo, Ontario-based company repeated a claim made earlier this month that it has a software fix that allows it to work around the disputed patents and keep the service going.

While there has been some fear that NTP could push for a BlackBerry blackout, few observers see that as likely. Many observers predict RIM will pony up a bigger cash payout to make the problem go away.

Lehman Brothers analyst Jeff Kvaal says RIM will likely seek a settlement to avoid an injunction and a disruption of service. "The amount of the deal is now likely to be higher than the prior $450 million settlement," writes Kvaal in a research note, adding that Wall Street will probably view anything under $1 billion as good news.

RIM has suffered a string of setbacks in the case over the last month. First, an appeals court declined the company's request to rehear the appeal in the patent suit before an expanded panel of judges. Following that, both the appeals court and the Supreme Court denied the company's request to delay further proceedings in the case until the Supreme Court decided whether to review RIM's appeal.

The latest setback for RIM is likely to be the one that forces a settlement in the long-running dispute, says David Schamens, partner with Invictus Funds.

"I think the market wants this uncertainty out of the way. If itis, the stock might really rip," says Schamens, whose fund has noposition in RIM.

Shutting down the service wouldn't be in anyone's interest, says Ovum's Entner.

"The worst case scenario is that existing users won't be forced to shut down, but there might be an injunction on selling to new subscribers," says Entner.

But NTP co-founder Don Stout says he will pursue an injunction if that's what it takes to get RIM to recognize its legal obligation.

"Here is a company that's a willful infringer and they don't want to pay us a dime," says Stout. "And if they refuse to pay us a dime, they are going to get shut down. We have no choice but to keep them shut down until they choose to take a license."

Jeff Neuburger, a patent attorney with Brown Raysman Millstein Felder & Steiner in New York, says Wednesday's court action "puts RIM in a very difficult position."

"They don't have a lot of options available to them," says Neuburger. From a legal perspective, "I don't see a lot of hope for them on the horizon," he says.

But RIM's statement Wednesday noted that the Supreme Court recently decided to hear an appeal in another high-profile patent dispute, that between eBay ( EBAY) and MercExchange.

Neil Smith, a patent attorney with Sheppard Mullin Richter & Hampton in San Francisco, points to that case. At issue in eBay-MercExchange is whether courts should by default issue injunctions against patent infringers, particularly in cases where the patent holder is a mere holding company -- not an operating business that actually uses the patents.

Even if the Supreme Court does not hear RIM's case directly,it could delay the enforcement of any injunction until it makes a ruling on eBay, Smith says.

The Supreme Court's decision to hear the eBay case "provides aray of hope [to RIM] of a substantial delay and a different result,"says Smith.

Jay Somaney, a hedge fund manager at TSG Capital Group whose firm has no position in RIM, expects to see a settlement but says the bill may be larger than investors expect. Somaney, whose firm sold its RIM puts Wednesday, believes RIM may end up paying NTP north of $1 billion on top of what he calls stiff licensing fees. He says those fees will force analysts to bring down their earnings projections for coming years.

"You buy on future earnings, and right now, earnings are not predictable," says Somaney. "At the moment there's too much uncertainty. And what we know thus far, going forward can only mean further bad news for RIM."

Schamens thinks the lawsuit has helped obscure other problemsfor RIM. Specifically, the competition is increasing in the company'swireless email business, helped along by new technology, such asfaster wireless connections, he says. That's why even if RIM's stockjumps on news of a settlement, Schamens thinks it is likely to fall as time passes.

The company recently had to cut subscriber-growthprojections for two coming quarters.

"Once you have bad news, there's usually more to come," Schamens says."I'm a firm believer in that. You don't want to be in these stockswhen bad news starts to come out after years of good news."

More from Technology

These 5 Tech Giants Still Aren't That Expensive

These 5 Tech Giants Still Aren't That Expensive

Intel CEO Brian Krzanich's Ouster Proves CEOs Aren't Above the Rules

Intel CEO Brian Krzanich's Ouster Proves CEOs Aren't Above the Rules

Amazon, Microsoft and Google Face Backlash over ICE, Military Deals

Amazon, Microsoft and Google Face Backlash over ICE, Military Deals

As Intel Loses Its CEO, How Well Can It Compete Against Nvidia?

As Intel Loses Its CEO, How Well Can It Compete Against Nvidia?

3 Great Stock Market Sectors Millennials Should Invest In

3 Great Stock Market Sectors Millennials Should Invest In