NEW YORK ( TheStreet) -- It's pretty simple: The political territory for Republicans led by John Boehner could be grim. The House speaker retained his leadership post on Thursday in a narrow vote that saw Boehner grab 220 votes -- just six more than what he needed. Conservative-leaning Web sites like Breitbart.com had warned ahead of the vote that Boehner would lose his speakership -- many others, including traditional media, predicted that he wouldn't -- the doubt spoke to a larger group of non-elected politicians who feared it would be the first domino of many leading to congressional losses in November 2014. "He's a disaster, we're falling apart," said one national Republican operative. "The party is falling apart; watch the next election, unless people do something they're not going to be able to keep control. We may actually lose in the mid-term -- control of the House -- and then it's all over." The worry among a growing number of Republicans is that Boehner's recent deal with President Barack Obama and Senate Majority Leader Harry Reid gave far too many concessions to Democrats and reneged on conservative promises to fight for bigger spending cuts and halt tax increases. It goes further back. So-called tea party conservatives fought the debt limit in the summer of 2011 because most of them ran on platforms that called for fiscal austerity -- not unlike measures seen in parts of Europe over the past few years -- and a promise to fight the very institution they worked for. With deficits mounting, and a number of analysts agreeing that the latest so-called fiscal cliff deal failed to cut into those deficits, Republicans who voted for measures that Obama called a balanced approach could find themselves in frigid primary battles against men and women touting many of the same promises that these incumbent conservative representatives made before. "We're going to have a lot of pressure here in Virginia, and I think our entire Republican membership in the House delegation voted against the compromise legislation earlier this week, which is a reflection that they're concerned about being challenged for the nomination," said Patrick McSweeney, former Virginia Republican Party chairman. "Every one of them faces a threat."