Updated from 1:40 p.m.

The BlackBerry patch got thornier Wednesday for Research In Motion ( RIMM).

A judge denied RIM's motion to enforce a previously reached patent settlement, ruling that a valid agreement was never reached. RIM shares fell 5% after being halted for two-and-a-half hours.

The stock has dropped 30% this year as Virginia's NTP has pressed a case claiming the BlackBerry messenger infringes on its wireless patents. RIM agreed earlier this year to pay $450 million to settle, but the deal fell apart in June.

The ruling leaves the fate of the BlackBerry with the federal court system and the U.S. Patent and Trademark Office. The case is currently at the District Court level after an appeals court largely upheld a ruling that RIM had infringed on several of NTP's patent claims.

"This opens the whole can of worms all over again," says Ovum analyst Roger Entner. "The refusal of the judge to enforce the agreement puts the whole settlement in doubt and brings them back to square one."

Investors had been hoping for some sign that an end was near for the prolonged legal battle. But now NTP is in a much stronger position and wielding the upper hand in negotiations. RIM shares fell $3.30 to $61.62.

RIM said in a press release that it will appeal the decision. The Waterloo, Ontario-based company repeated a claim made earlier this month that it has a software fix that allows it to work around the disputed patents and keep the service going.

While there has been some fear that NTP could push for a BlackBerry blackout, few observers see that as likely. Many observers predict RIM will pony up a bigger cash payout to make the problem go away.

Lehman Brothers analyst Jeff Kvaal says RIM will likely seek a settlement to avoid an injunction and a disruption of service. "The amount of the deal is now likely to be higher than the prior $450 million settlement," writes Kvaal in a research note, adding that Wall Street will probably view anything under $1 billion as good news.

RIM has suffered a string of setbacks in the case over the last month. First, an appeals court declined the company's request to rehear the appeal in the patent suit before an expanded panel of judges. Following that, both the appeals court and the Supreme Court denied the company's request to delay further proceedings in the case until the Supreme Court decided whether to review RIM's appeal.

If you liked this article you might like

Apple iPhone 5 Gets Oct. 4 Launch Date

U.S. Moves to Block AT&T/T-Mobile Deal

Sprint Will Start Selling iPhone This Fall

A Cellphone With an Airbag Might Be on Its Way

Google+ Halfway to Critical Mass