Big data bringing Asimov's Psychohistory from fiction to reality
Isaac Asimov, my favorite science fiction writer and thinker introduced a concept called Psychohistory in his famous “Foundation” novel series.
Psychohistory is the science of predicting the behavior of large groups of people using a statistical model of the psychology of groups using historical data. And for events in the future are primarily the outcome of the what people do, by predicting the behavior of people one can effectively predict the future!
There are some logical requirements or assumptions that are the basis of Psychohistory:
- The population of the group should be large enough
- The amount of data available about the behavior of people should be sufficiently detailed
- The technology to process such large and complex data is available
In the last 60 years since Asimov first introduced Psychohistory in 1950’s, all these criteria are starting to inch closer to reality.
- The population of the earth has crossed 7 billion – higher than the 2.5 billion – the population in 1950, but still quite small as compared to the “twenty-five million worlds…each containing a billion or more human beings” (Thanks Bob for the correction)
- The democratization of data with the availability of cheap data creation tools such as mobile phones has resulted in a huge explosion of data available about people’s behavior.
- And the third phenomenon of the technology to process such large and complex data is now available.
As a result, it is becoming increasingly possible to make predictions about human behavior based on such data.
Take the example of an interesting experiment done by a group of mathematicians and computer scientists in 2010 based on documents about the war in Afghanistan that were made available via Wikileaks. They extracted historical patterns of violence from these documents and could effectively predict where future events of violence would occur in Afghanistan.
Then there is the famous example of Andrew Pole, an economist and statistician working with the retail chain Target who mined historical buying data to predict which customers had a new baby on the way. The article in NY Times quoted a sensational anecdote of how the pregnancy of a teenage girl was predicted before her parents knew about it!
Another sensational story is of Dan Wagner, the chief analytics officer for the Obama 2012 campaign, who led the campaign’s “Cave” of data scientists. They used data analytics and the experimental method to assemble a winning coalition vote by vote. In doing so, they overturned the long dominance of TV advertising in U.S. politics and created something new in the world: a national campaign run like a local ward election, where the interests of individual voters were known and addressed.
These examples clearly demonstrate how large enterprises and governments are able to use historical data to predict specific outcomes in the future, making it appear closer and closer to Asimov’s Psychohistory.
But there is one thing that Asimov probably did not foresee – he imagined that Psychohistory would be used by a central agency that would predict future events so as to effectively govern the people. However, what is actually happening now is that the ability to analyze historical data is becoming accessible to everyone, and is not restricted to only governments or large organizations with big budgets.
Consider these facts:
- A large part of the data that represents human behavior is available to everyone via the Internet (social media, blogs, email, documents)
- The technology required to process this data – both software and hardware is affordable (cloud computing) and mostly free (open source)
So it is possible for smaller entities with low budgets to use data to predict outcomes and plan their actions to achieve fruitful outcomes.
Of course, what it takes is some smart people to know how to use this data and technology and get meaningful insights – that’s the mission that (smart?) people like us are trying to actualize 🙂
