Thinking in Reverse

Many of the benefits offered by new ways of working require you to think in reverse. Adapting in real-time rather than sticking to plans 

At the end of June, I shared a link – The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. Here’s the short version of what’s in the article:

Correlation between data does not guarantee causation. If X increases when Y increases, they share correlation. But we don’t know that an increase in one causes an increase in the other. It could be that something else – Z – causes an increase in both. Because of this, we need to create a model – a hypothesis – and then use data to test the model. It’s the way science works.

Or, rather, it’s the way science has always worked until now…

With the increasing volume of data available, the article argues that using a hypothesis is fast becoming obsolete. That, in the face of massive amounts of information, correlation is indeed enough. Let the computers find patterns in the data. Unsurprisingly, Google is in favour of this approach 🙂

Because all of my shared items end up on my FriendFeed page (along with blog posts and Tweets), when I first shared the link to this article, it was picked up by others and a conversation started about it (you can view it here, third link down at time of writing). Everyone, including me, was against the idea of abandoning models. But then, last week, I read a similar article in the New Scientist – Crunchonomics (UK print ed) – and I’m beginning to change my mind.

My initial thought when reading the Wired article is that to assume causation from massive correlation is risky because it involves assumption. But if you flip the viewpoint and question how confident are you that your hypothesis is correct just because the test data agrees with it, we’re no better off.

Crunchonomics points the finger of blame regarding the current state of financial markets at out-of-date theories about how markets work. When assessing market risks, regulators rely on ‘equilibrium theory’: the hypothesis that market values change only in response to external influences that take them away from their stable equilibrium state. Rumours and speculation are not recognised as external influences yet they undoubtedly disrupt markets, causing deviations that are unexpected (don’t fit the model) and hard to explain. In response, researchers have been trying a new approach: “agent-based models”. These models do not assume a hypothesis from the outset. Instead, they let market behaviour emerge naturally from the actions of interacting participants. They simulate a possible scenario (e.g. raising interest rates) and observe what happens. And they are already outperforming the traditional models.

An example was published in the New Scientist magazine a couple of months ago. Traditional financial models used the assumption that people will sell off lower cost items before risking losing their house. So a trend in missing repayments of car loans would be an early warning. But that hasn’t happened this time (in the US). Thanks to negative equity, people are giving up on the house before the car. A simulation would have spotted this new pattern emerge. The traditional hypothesis wasn’t even looking for a new pattern to emerge and ignored the data until it was too late…

The challenge with establishing this new approach to prediction is validation. If you’re simply observing what’s happened in your simulation, and making predictions based on the outcomes, people will question it’s credibility. Where’s the nice tangible hypothesis, you’re telling me ‘it just happened that way’…

And maybe that’s the challenge we face, to think in reverse. To stop dismissing new ideas because they are polar opposites to the old ones that we understand and are comfortable with (no matter how often they fail). In the serendipity that is the web (and especially Twitter these days), I came across a blog post by Sue Thomas about Richard Feynman. Sue embedded a video that included a comment we should all consider living by:

“People say to me, ‘Are you looking for the ultimate laws of physics?’ No I’m not. I’m just looking to find out more about the world. And if it turns out that there is a simple ultimate law that explains everything, so be it. That would be very nice to discover. If it turns out that it is like an onion with millions of layers and we’re just sick and tired of looking at the layers, then that’s the way it is. But whatever way it comes out, it’s nature…”

If abandoning hypothesis gets us to open our eyes more to other possibilities, maybe all this data will come in useful 🙂

What’s particularly interesting to me is that this challenge goes way beyond disrupting a hypothesis. Many of the benefits offered by new ways of working (under the labels Web 2.0 and Enterprise 2.0) require you to think in reverse. Prediction markets challenge the belief that strategy should come from management. Open source collaboration challenges the belief that research and development must be performed in secret. Blogging challenges the belief that PR and marketing control external communications…

I suspect the next generation of leaders will be people who are comfortable with thinking in reverse. It’s not about ignoring the past and throwing all the models out of the door – a dangerous thing to do. Rather, question if the past was as successful as it could have been…