I’ve been bloviating about predictive enterprise analytics for more than a year now, discussing it with clients, vendors, my mom, the kid who mows my lawn, and anyone else who will listen. I think it’s going to be the ‘next big thing’ in business, and thus enterprise technology. The last time I saw a tidal wave this huge on the …
Does this mean that a move towards the business intelligence space would be a good career shift for UK IT pros? It seems to be already rather heavily populated by graduates of the Indian universities, encouraging the off-shoring model.
While I think that more BI background would help IT pros in the UK (and elsewhere too), I think that they'd be better served by learning more about predictive analytics. By this, I mean learning more about modeling, data analysis, and predictive methods - lots of statistics. But this knowledge needs to be coupled with business expertise and curiosity. The most valuable folks will be those who can apply the analytical techniques to the unique business situation their organization faces. Machines can do the math and crunch the data, but it takes smart people to figure out the right questions to ask, the right conditions to test, and the right data to use to get the answers. In my mind, these skills are the most valuable and will be very difficult to outsource or offshore.
Most businesses already have a strategy
You'll have a tough time ttrying to sell a non-intuitive conclusion to a CEO - no matter how many scads of data you have "processed". If you can't present a consistent (and simple: very, very simple) argument about why a particular strategy will bring huge returns, there's little possibility that they'll risk their career and bonus on something that doesn't appear to make sense.
Most companies have a very straightforward and well understood business strategy: find out what the competition's doing and do the same. It's not so much about making huge profits, as avoiding huge losses. While no CEO ever got fired for doubling the dividend, not many get fired for making a loss - provided all the other, similar, companies are doing the same.
Modelling - one industrial perspective
You might find http://www.hpl.hp.com/techreports/2003/HPL-2003-246.pdf of interest.
smart data dumb tools
I like it .. so HPC is like getting a more powerful power tool. As you correctly identify the quality of your shelves is determined by the lowest common denominator in the chain of tools in the process.
Similarly with enterprise analytics the ability to process large amounts of data is not likely to be the limiting factor. The reason that fluid dynamics is such a good application for HPC is that there is a well understood model that is computationally intensive.
Trying to find the same in a business context leads me to madness or down the route of dealing systems based more on game theory than predictive modelling.
Corporate IT has kindly enabled me to pull enormous volumes of data going back 10 years to support the development of my regression models, and boy are we pleased with ourselves. Technically I can build an effective predictive model from a few thousand records and it will be just a useless as one built using the whole data warehouse. 10 years of historic data tells me nothing in a world/market/game that is constantly changing. The past is not a good indication of the future, the markets constantly change and the game landscape (other players actions) constantly change.
The challenging is to get accurate, fast feedback or measures…. and to make quick decisions based on this in order to react faster than the competition… and the chain of tools that must be employed is the corporate landscape is looooong.
I can build simple adaptive models that react to the changes that I suspect would be very effective but I don’t have reliable real time data measured at point of delivery, (ok and I have problems with good evaluation functions). Furthermore I don’t subscribe to the view that you need to waste time extracting the data.. why hawk data out of the environment it is collected in when we can embed adaptive models in the front line systems. But this is not likely to happen as we in IT continue to ‘solve problems’, ‘meet requirements’ based on the premise of data warehouses and the need to process large amounts of data.
“Smart Tools…Dumb Data” …. aaargh they have go me at it now.
Sorry that should of course have read give me “Smart Data…Dumb Tools”
I work for a company that does precisely that, produces software to embed adaptive models in front line systems. When you add real-time visualisation and the ability to change the rules in the system more or less instantly you have a very powerful tool for controlling your business. Works particularly well in banks, telcos, utilities,etc. Don't knock predictive analytics too much, it can be a useful tool, as long as the results aren't treated as Delphic.
used it, need to use more
I've used Data mining and forecasting in SQL Server just last year and have more than 10 years experience with databases, not sure if it says something about me or about how much managers understand technology.
It actually provides you with 6-7 different algorithms to do your predictions & data mining. I was actually quite surpsised how I never explored this option before. I am not familiar with other tools but I can see a huge growth in this area. Business just strated to wake up to multi dimensional databases and data mining. I hope it will go this way..
Those aren’t easy questions to answer, particularly by an amateur economist.
The average guy reading this will have an IQ over 130 and ranging up to 170, the average woman reading this, over 115 ranging up to maybe 150, and the average economist?
To find out who knows what they're talking about, you simply have to identify all the economists who've made a mint buying puts because they predicted the collapse, and have since retired to Provence. Oh. that's right, there aren't any. They're all still putting their names to letters saying how good Gordon Brown's policy of tax and spend is, just like they did last time.
I once ended up in Bermuda in 91, at a wedding with a guy who used to work for the IMF in New York, discussing my ideas on the domino effect that was occurring out of the far east, having just spent the previous day on a fantastic course (92 - drove 340 yards on the first,) with a high ranking fed official, who told me in confidence that the Fed weren't going to do anything substantial and were going to let it run its course, (although this was all pure co-incidence, my wife booked the holiday, she was bridesmaid.)
We talked about economics, and he said, economic theory was so simple that by the mid 70s you could model it with flowing water and pipes. That was, he noted, until Save the Whales came along. After that, he concluded, economics became merely a side effect of the shift of human behaviour, public opinion could simply block a hole, like Nestle's problems, or Barclays in ZA.
The way to forecast is therefore to get someone directing your business who is astonishingly clever, but is also a cynic because that's what customers are.