Editor's note: This is a special bonus column for TheStreet.com readers. Anirvan Banerji's column appeared Oct. 31 on RealMoney.com. To sign up for RealMoney, where you can read his commentary first, please click here for a free trial.

In the early summer, I

highlighted the structural shift in manufacturing jobs, which account for 90% of U.S. job losses since the recession began. Back then, it was a fresh -- and controversial -- insight, on the basis of which I predicted a lopsided recovery, more in GDP, less in jobs. Today, it's not just a well-known fact; it's a hot-button political issue.

Growth in the Economic Cycle Research Institute's

Weekly Leading Index soared to a 20-year high in the summer, raising many eyebrows. Three months later, GDP growth has rocketed up to a 19 1/2-year high, raising fewer eyebrows because the earlier pessimism has largely waned.

The Red Queen Effect

Back in the summer, the ruling consensus held that this was a jobless recovery like the one in 1991-92, and strong job growth would show up once GDP growth increased. But the structural shift is rooted in firms' lack of pricing power, forcing them to cut costs by boosting productivity and outsourcing jobs overseas -- a dynamic that's still unchanged. Therefore, while job growth will surely improve as GDP growth ramps up, it will have to contend with the structural headwinds impeding job creation -- what one might call the Red Queen Effect.

As the Red Queen in Looking Glass Land tells Alice, "Now, here, you see, it takes all the running you can do to stay in the same place. If you want to get somewhere else, you must run at least twice as fast." What policymakers are therefore trying to do is to make the economy run twice as fast, and that will indeed help job creation. But because of the Red Queen effect, job growth still won't match GDP growth.

Why did so many analysts fail to realize that a structural shift was under way? Why did they fall en masse for the analogy with the 1991-92 recovery, missing what was truly different about this cycle? The answer has a lot to do with many analysts' basic approach to forecasting.

Forecasting by Analogy

Analysts often assume that because the present resembles earlier periods in some way, it does so in other ways. They then predict that the current period will exhibit similar patterns.

If the chosen analogy doesn't work, they hunt around for other analogies -- other periods this might better resemble. And if none of their analogies work, they proclaim a "new paradigm."

How many times have you heard that home prices always fall during recessions? Or that September is a bad month for stocks? Or that stock prices track the ups and downs of hemlines? Or that sustained above-trend GDP growth causes inflation?

Well, we saw in the late 1990s that, contrary to all the forecasts by analogy to earlier experiences, inflation stayed low in spite of years of robust growth. That's when many economists invoked a "new paradigm" of endless inflation-free growth.

Many "sophisticated" econometric models are based on similar logic. In the late 1980s, a pair of econometricians gained prominence with their recession probability model, where they fitted a bunch of indicators to a Vector Autoregressive model and derived recession probabilities from a Monte Carlo simulation on the results (never mind)!

The point is that they found the strongest linear association based on past data, and assumed that precisely the same relationship would hold in the future. But there was never a good answer to the question, why? In truth, this is nothing but a fancy example of forecasting by analogy.

When their model failed to predict the 1990-91 recession, they explained that the culprit was "parameter drift." In other words, the model parameters they had estimated, based on past data, had changed. So they rejiggered their model. Unfortunately, it failed to predict the 2001 recession as well.

Long-Term Recession Prediction

The pitfalls of forecasting by analogy may be best shown by an example from experience. A decade ago, I worked with ECRI's founder, Geoffrey H. Moore, on ways to predict recessions years ahead.

TheStreet Recommends

There had long been research showing a link between two key variables: the lag between the business-cycle trough and the trough in interest rates; and the remaining duration of the business-cycle expansion. In other words, how long it took for interest rates to turn up after a business-cycle bottom could provide a clue to the timing of the next recession.

When we used the prime rate in this model, we found something surprising. The lag in the prime rate at business-cycle troughs had a very strong relationship with the remaining length of the expansion.

Source: ECRI

In fact, the variables had a correlation of 95% -- virtually unheard of in economics. And this uncanny relationship had held for four decades after the 1951 accord between the U.S. Treasury and the

Federal Reserve

, when the prime rate started fluctuating after being fixed for years around World War II.

We understood why there was a general relationship. The puzzle was why it was so remarkably tight, and we never could figure it out. As a result, we never tried to publish the results in an academic journal, fearing that the model would break down. Nor did we try to use it to predict a recession. Without knowing why the relationship was so neat, using it to make a prediction was nothing but forecasting by blind analogy to past patterns.

But Moore couldn't resist privately sharing the findings with an old student, then the Fed chairman, who responded that he sure hoped it worked this time. Mind you, the year was 1993, there were still lingering fears of a double dip, and our prime-rate model was predicting that the expansion would last at least three more years.

But the long expansion of the 1990s broke the mold. An expansion that, according to the prime rate model, should have ended by late 1996, morphed into a boom that kept going until the new millennium.

Source: ECRI

The Neat Little Model That Couldn't

This is the first time we've publicly shared the story of our neat little model that couldn't. It's a cautionary tale that should put people on guard against the distressingly common practice of forecasting by analogy. Bottom line: There are tantalizing relationships suggested by data mining, but it's reckless to base forecasts on them without knowing just why those precise relationships should continue to exist.

This also illustrates a basic principle of the work we do at ECRI, which is limited to relationships that we do understand. We don't base our forecasts on pretty patterns -- no matter how compelling -- because we can't trust them in the absence of clear economic logic for their existence.

But the prime rate model does suggest that a prolonged period of low interest rates may result in a long expansion this time -- unless the model breaks down once again!

Anirvan Banerji is the director of research for the

Economic Cycle Research Institute, which was founded by Dr. Geoffrey H. Moore, creator of the original index of leading economic indicators (LEI) for the U.S. Department of Commerce. Banerji is on the economic advisory panel for New York City, and is also a member of the OECD Expert Group on Leading Indicators. At time of publication, neither Banerji nor his firm held positions in any securities mentioned in this column, although holdings can change at any time. Under no circumstances does the information in this column represent a recommendation to buy or sell stocks. While Banerji cannot provide investment advice or recommendations, he welcomes your feedback at

Anirvan Banerji.