What is the difference between predicting an election result and a stock movement?
Actually not that much, for both involve one fundamental ingredient that has always dogged the social sciences. Both involve an assessment of how large numbers of humans will behave at a given time. And unlike photons, humans do not rigidly follow strict laws of nature. They are in themselves very complex systems that are likely to change their views quite suddenly, be influenced by exogenous events or be plainly irrational.
This becomes particularly apparent when we come up to the results of an election as tight and fraught as the current U.S. presidential election. Trying to figure the end result becomes more akin to predicting short-term currency movements, which often seem somewhat random.
And just as with stocks, vast data are provided in elections, along with multiple methodologies for assessment of the data. The biggest volume of data is, of course, polls. But polls do not necessarily reflect how someone will ultimately behave when they are alone in that booth making their final voting decision, nor what factors can arise between the time of the poll and the actual vote. The selection of polling participants can also cause bias in results.
Then there are those who tell us they have systems. There is the "13 keys"-based analysis of U.S. history Professor Allan Lichtman, or the recent artificial intelligence system that predicted correctly the primaries and the last three presidential election results. Or there's the stock market itself. Going back to World War II, when the S&P 500 has declined between between July 31 and Oct. 31, a challenger has won the presidential election 86% of the time. The S&P 500 is down about 2.0% since its close on July 29.
But just as financial past performance is always only a guide to the financial future, likewise past political performance and correlations too can only be a guide. Over long periods, historical drivers also can fundamentally change as society changes. Even Professor Lichtman's history of predicting the last eight election results in a row correctly may not be statistically significant. If a fund manager comes to me telling me he has had decent positive returns for eight years, does that really mean that much given how ubiquitous fund managers are today? In other words if 50,000 people are tossing coins, just by the natural laws of probability, some will get eight heads in a row.
So we have a pretty deep dichotomy in this type of predictive analysis. The very fact that it is happening at the level of the individual human is intrinsically problematic, because the individual human is itself a highly complex thinking and emotional system. All of these predictive analyses gloss over much of (all of) the complex inner mechanics of the human mind (whether it's a voting decision or the decision to buy/sell a stock).
It seems hard that we will ever be able to overcome this problem until the day we know much more about the chemistry of the brain itself. Only then perhaps could we somehow try and compute these collective decisions based on very complex brain chemistry patterns across thousands/millions of people. For the moment that is largely science fiction.
For now, it seems, the best we can hope for is methodologies that perhaps provide us with reasonably reliable "modal" type outcomes -- i.e., based on the last say 50 years of behavioral patterns it is at least more likely that candidate A will win rather than candidate B.
It is one of the critical reasons economists and political theorists should be careful about calling their disciplines "sciences" when they don't seem able to replicate the type of high-level certitudes we can extract from, for example, physics or chemistry.
So, will it be Clinton or Trump? We had better leave it in the hands of destiny.