Expecting a rebound for luxury retailers? Yeah, you might want to reconsider that position. The sector is the weakest among retailers. On Wednesday privately-held Neiman Marcus said it swung to a loss in the third quarter of $3.1 million, compared with a profit of $55.4 million in the prior year. Revenue fell to $810.1 million from $1.06 billion. Last month, Polo Ralph Lauren ( RL) reported that profit in the fourth quarter was more than halved on increased discounts, and Tiffany & Co. ( TIF) earnings plunged 62% in its first quarter. May same-store sales in the sector also saw some of the biggest declines. Saks ( SKS) posted a 26.2% plunge, while Nordstrom ( JWN) sank 13.1% during the month. This weakness is not expected to slow near-term. According to a report today by Claudia D'Arpizio, partner and luxury consultant at Bain & Co., high-end department stores and specialty retailers will not fully recover until 2012. D'Arpizio forecasts a 10% drop in worldwide luxury spending in 2009, after spending was flat last year and progressively worsened throughout the year. She foresees sales remaining at 2009 levels in 2010 and then rising 4% in 2011 and gaining 7% to 8% in 2012. But what has been scaring investors the most is the fear that even after the economy recovers shoppers will be hesitant to return to their old lifestyle. Retailers like Wal-Mart Stores ( WMT) have been claiming that they will retain shoppers who have traded down during the recession, even when confidence returns to the economy. But what does this mean for the traditional high-end retailer? Well, D'Arpizio says the long-term prospects for the sector remain strong, as long as retailers invent better in-store experienced and find ways to target the younger shopper and working women.