back to article Who you callin' stoopid? No excuses for biz intelligence's poor stats

Business Intelligence (BI) systems are designed to turn raw data into useful information, so why don’t they do the job properly? Why do most of them fail so completely to make use of the huge range of capabilities that the analytics world has to offer? Even at the most basic level, they fail catastrophically to take simple …

  1. AMBxx Silver badge

    Money!

    It's easy enough to do proper stats with proper data with failry simplistic software. No money in that.

    Much more money in data exploration - lots of hardware and fancy demos, plus a good buzz word in 'Big Data'.

    In many ways we did this stuff better 10 years ago when infinite hardware resources weren't available.

  2. Anonymous Coward
    Anonymous Coward

    In many ways BI is going backwards.

    Since many bussiness-men (and women) will see stats as a tech-problem where you should give someone computers and files and then the techie will give you answers the quality is steadily declining.

    If bussinesses really wanted to improve they would start with the basics (i.e. do the basic stats mainly confirm what we think we know) and then add on increasingly complex reporting.

  3. Efros

    Multivariate analysis

    Arrrrrrrrrrrrrghhhh!

  4. Primus Secundus Tertius

    Inspector Nectar

    Is it stupidity or perversity?

    My nectar card regularly receives vouchers for immense discounts if:

    • I buy far more than I want to buy
    • I buy things I would never dream of buying.

    I wonder why I bother.

    1. Dan 55 Silver badge

      Re: Inspector Nectar

      I'd say it's working. Why would they give you a discount on stuff you always buy anyway, that would make them lose money. They want to make you buy much more of the same stuff or new stuff, that's where the money is. And if you were to, say, regularly start buying something they originally offered you a discount on, I bet that offer would disappear fairly soon too.

      1. Anonymous Coward
        Anonymous Coward

        Re: Inspector Nectar

        I don't think that is always the case.

        I bought a few copies of the 'I' for a neighbour recently.

        Lo and behold, I got an offer of 40 Nectar points the next time I bought a copy of the 'I'.

        The same goes for coffee.

        1. Anonymous Coward
          Anonymous Coward

          Re: Inspector Nectar

          Lo and behold, I got an offer of 40 Nectar points the next time I bought a copy of the 'I'.

          The same with Amazon and most on-line retailers. Particularly for Amazon, given the amount of data they have, and the vast processng power at their command, I marvel at the stupidity of their "recommendations".

          If the junk offers from loyalty cards, and recommendations of on-line tat retailers are the height of "Business Intelligence", then I think that we can safely conclude that big data is in fact a big fat blind alley.

          1. Dave Bell

            Re: Inspector Nectar

            Amazon regularly lies with statistics.

            They talk about "averages", and there at least three that can be used, all simple and each giving different results with their "long tail".

            1: Mean

            This is the one that most people use if they have to give a figure. 100 samples, add the numbers together, and divide by the number of samples. But a few high figures, such as a millionaire when everyone else is on minimum wage, will make that average useless. There are authors who sell thousands of books for the Kindle. Are you that good?

            2: Median

            When the samples are arranged in order, low to high, this is the one in the middle. This is closer to our idea of "average". That one-millionaire problem obviously doesn't affect the figure.

            3: Mode

            You split the samples into groups, like that age chart in the article, and the mode is the biggest group. You have to be careful about it, because changing the size of the group can shift it. The mode for those Kindle sales is effectively bugger all. That's what the "long tail" means.

            They're teaching this in the schools now.

            1. Rich 11

              Re: Inspector Nectar

              They're teaching this in the schools now.

              They were teaching this in schools back in the Dark Ages (well, the 1970s), though I recall Biology as being the subject which used the most illuminating practical examples rather than Economics.

  5. Tom 7

    Pearls to swine!

    I have had the misfortune to work for Managers who proudly profess they dont need to understand what they are managing, merely manage it properly. I have worked with a senior accountant where I have shown him how to use Newton-Raphson to minimise the cost of a relatively simple process.

    Two things will have to happen for BI to catch on properly

    1) Those in sales looking at the output will have to learn what it actually means and that will require more than a C at O'Level maths.

    2) Managers will have to actually know more about it than those in sales as this will have to be applied to production etc.

    Neither is going to happen in my lifetime.

    1. Uberseehandel

      Re: Pearls to swine!

      You might have more success had you taught the accountant how to formulate his optimisation problems using linear programming techniques, and, importantly, how to interpret the dual. If find it hard to imagine a qualified accountant in industry who does not understand how to do this, but then I got my qualifications abroad.

      Few managers in industry, or senior treasury officials have any concept of the time value of money, either.

      Applied Statistics, Applied Mathematics and Operations Research all benefit from charismatic lecturers, unfortunately, not commonly encountered.

      I have no idea what gets taught in Data Analytics these days, and few people have any concept of what an Informaticist does.

      For many organisations, BI is handled by web developers. Who most definitely are not the right people for this task.

      1. Rich 11

        Re: Pearls to swine!

        For many organisations, BI is handled by web developers. Who most definitely are not the right people for this task.

        Speaking as a web developer who qualified in maths and statistics... no, you're right. I've forgotten far too much to trust myself to do more than spot problems rather than offer solutions.

    2. keithpeter Silver badge
      Coat

      Re: Pearls to swine!

      "1) Those in sales looking at the output will have to learn what it actually means and that will require more than a C at O'Level maths."

      GCE O Levels had numerical grades for most of the life of the qualification, grade 1 being high and grade 9 being 'should have done French'.

      New GCSEs next year also numerical grades but big endian so grade 9 high and grade 1 is 'should have done pet husbandry' or 'I have dyscalculia'.

      New GCSE grade 5 or 6 will be around what you are implying in a couple of years as the changes work their way down the school (image: house being built from roof down, roof suspended by a couple of Chinook helicopters in the interim).

      Moral: As Slackware LILO puts it when you have done something stupid 'expect trouble ahead' on Maths in UK for the next few years. Also English, and then the Sciences.....

      Heads up for employers of technician level staff: Look out for Core Maths. This is a Level 3 qualification aimed at those who passed their GCSE and went to the local FE College. Has modules on decision maths (graph theory), stats (significance tests) and Proper Algebra. Those youngsters might be handy.

      1. Doctor Syntax Silver badge

        Re: Pearls to swine!

        "house being built from roof down"

        This is very successful as it means the builders can do the rest of the work in the dry.

    3. Vic

      Re: Pearls to swine!

      I have had the misfortune to work for Managers who proudly profess they dont need to understand what they are managing, merely manage it properly.

      I firmly believe that such a creature is possible; a sufficiently-skilled manager could get al the technical know-how from those working for him as required.

      I've never met one, though...

      Vic.

      1. Anonymous Coward
        Anonymous Coward

        Re: Pearls to swine!

        "I have had the misfortune to work for Managers who proudly profess they dont need to understand what they are managing, merely manage it properly."

        Standard behaviour for PMs at our place ... if you do the other thing, seems to be a bit career-limiting .....

        "I firmly believe that such a creature is possible; a sufficiently-skilled manager could get al the technical know-how from those working for him as required. I've never met one though" .. I have. He was a cat herder, worked down near the unicorn farm.

    4. Voland's right hand Silver badge

      Re: Pearls to swine!

      senior accountant where I have shown him how to use Newton-Raphson

      Accountant? With knowledge of differential equations and/or optimal control?

      Dude, you are smoking something cool which in itself is not a crime, but you are not sharing. That is criminal.

    5. Sam Adams the Dog

      Re: Pearls to Swine

      Neither is going to happen but neither is necessary. As the article makes clear, it is not Sales or Management that has to understand the math. It's the planners and developers of BI software, who presumably can understand the math and can translate the conclusion into fairly simple assertions.

      1. MachDiamond Silver badge

        Re: Pearls to Swine

        "Neither is going to happen but neither is necessary. As the article makes clear, it is not Sales or Management that has to understand the math. It's the planners and developers of BI software, who presumably can understand the math and can translate the conclusion into fairly simple assertions."

        I've always found that it's a very good idea to have a basic understanding of a process so I know when I'm being fed a length of rope. It's the same as approximating an answer on a maths quiz as a check on whether you're calculation is even in the same county. Ever slipped a decimal when using a calculator?

    6. MachDiamond Silver badge

      Re: Pearls to swine!

      The managers that profess that they don't need any knowledge of what they are managing is a byproduct of going to B school. I've worked with these types that have had no idea about the products and the culture of the marketplace they sell in. It leaves me convinced that part of the entrance boards for an MBA is to hold a certificate that a set minimum amount of brain was excised during their lobotomy.

  6. Vaidotas Zemlys

    Learned any statistics lately?

    Talk about facepalm. The author argues that BI should give more information to the users, and then gives the wrong p-value interpretation. It is especially ironic, as recently American Statistical Association made a statement of interpretation of p-values (http://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108). In that statement the list of wrong interpretation of p-values is given and the second statement is the following: "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone".

    Yes, there are a lot of things where BI could improve. Hopefully not in the direction of claiming things that are not simply true.

    1. Penelope Gwendolyn

      Re: Learned any statistics lately?

      You are quite correct, the paper you quote does, indeed, say that "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone".

      One of the points that this statement is making is that factors other than random chance can affect a result. This is also perfectly true. And if we had complete knowledge of the situation we would also know those other factors and (if we were smart enough) take them into consideration. But if we don’t know any of those, then the statement given in the article “There is a 25 per cent chance that the difference in these figures is simply due to random chance.” is the closest estimate that we have.

      It is also worth noting that the paper you quote also says “Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.” This, informally, agrees with the statement given in the Register article.

      However this very discussion we are having simply highlights that the field of statistics is complex and open to interpretation. Indeed, to again quote from the same you cite:

      “On behalf of the Board, he reached out to over two dozen such people, all of whom said they would be happy to be involved [in creating an ASA statement on p-values and statistical significance]. Several expressed doubt about whether agreement could be reached, but those who did said, in effect, that if there was going to be a discussion, they wanted to be involved.

      Over the course of many months, group members discussed what format the statement should take, tried to more concretely visualize the audience for the statement, and began to find points of agreement. That turned out to be relatively easy to do, but it was just as easy to find points of intense disagreement.”

      So the experts assembled to create the paper from which you quote clearly disagreed. This is perfectly reasonable, as I said above, the field is complex and even eminent statisticians disagree about interpretation.

      I believe that the author of this Register article was making the point that most people:

      do not have the required training to interpret statistics accurately.

      are not interested in the finer detail.

      If that is the case, our options are to:

      continue not to inform the business user of the availability of statistical analysis.

      use statistics to give the business user SOME guidance.

      So, by telling the user that “There is a 25 per cent chance that the difference in these figures is simply due to random chance.” we are giving them the best guidance that we can, given the information we currently have and certainly more than by giving them a bar chart.

      We can argue until we are blue in the face about the wording that should be used but we need to bear in mind that, if it is made too complex it will convey no information and we will be back at square one.

    2. Gordon 10

      Re: Learned any statistics lately?

      @vaidotas

      Google the author (yes he is the top hit). Having been taught by him its a fair bet that he knows a lot more about Stats than you.

  7. glen waverley
    Pint

    what i remember from uni stats ...

    The only thing that I remember for sure from the stats units i did at uni is this ...

    The man who developed student's t-test worked at the Guinness brewery.

    Not quite the right icon but near enuff.

    1. keithpeter Silver badge
      Coat

      Re: what i remember from uni stats ...

      "The man who developed student's t-test worked at the Guinness brewery."

      And Guinness didn't want people knowing they employed statisticians and did proper process engineering in their huge factories (in the 1930s!) so they had him publishing under a pseudonym - hence we have Student's t-test.

      Google for the paper by 'Student' about the Lanarkshire Milk Experiment for some fun reading on a bank holiday. Big data from before the war...

      Moral: a result can be statistically significant but not really very meaningful. Especially with huge numbers in your samples. Just how much do you want to spend to sell 1.6% more oranges to the fictional ladies in the OA?

      Coat icon: its raining.

      1. glen waverley

        Re: what i remember from uni stats ...

        Interesting reading. Thanx for that.

        My takeaway: another example of the maxim that no plan survives contact with reality! As in letting the teachers pick the kids for the supplementary milk group.

        (And the rich (less poor?) kids lost weight as they shed their winter clothes over the duration of the experiment.)

  8. Anonymous Coward
    Anonymous Coward

    A BI-pusher writes...

    I work for one of those BI vendors - probably one who would earn a bad report card from the author - and I see plenty of what customers' actual buying priorities are. And because buying priorities are what pays for for our high salaries and extensive perks (or senior management bonuses - one of those), they tend also to be reflected in the product managers' priorities.

    The want: Cloud (because... cloud), mobile, interactive visualisation (nb. actual quality of information conveyed by the visualisation is considered as a nice-to-have rather than necessity), fast performance (because nobody wants to wait for anything), self-service authoring for end users, a modern UI with as much wasted screen real estate as possible, integration with whatever technological fad their CIO has read about, and so on. Statistically thorough information just doesn't seem to appear on the list, which is ironic given that we also have some great statistics software.

    I agree that quality and usefulness of information conveyed by analytics software is crucially important, but it appears to be surprisingly unimportant to many of the prople looking to buy stuff. If we want to sell product, we instead need to meet all these other needs which are somehow much higher up the list and well, our development bnudget certainly isn't getting any bigger (unlike the chief exec's bonus).

    On the flip side, I've also met plenty of data-science types at various customers and potential customers, and they too often have a somewhat blinkered view and tend not to place enough value on aspects such as a good UI, a system that is responsive to changing needs of users, ability to scale a system into production, data stewardship and so on. A statistically excellent spreadsheet that sits on some crufty network share that only four people in the company outside of the data science department actually know about and may or may not contain up to date, accurate or canonical numbers is possibly not the greatest thing either.

    1. gregthecanuck

      Re: A BI-pusher writes...

      I cannot upvote your comment enough times.

    2. a_yank_lurker

      Re: A BI-pusher writes...

      Upvote for telling truth. Most of the problem is people want sizzle not steak. If it looks good and sort gives something interesting it must be good not whether the "analysis" is better suited for fish wrap than for decision making.

  9. Howard Hanek
    Headmaster

    Stoopid?

    No. That whole slavish devotion to Samuel Johnson/Dictionary thing is stoopid.....

  10. Vic

    How rhetorical is this article?

    Are the answers really not obvious?

    Modern Management™ uses statistics as a drunk uses a lamppost; more for support than illumination.

    Thus no-one is going to go to the big boss with an idea when it's clear there's a moderate-to-strong chance that the "inspiration" found in a data set is actually just random chance. And no middle-manager is going to want to justify to his boss why his output for the month is no more and no less than was churned out by the BI tool .

    People want tools that make them look good. They don't want tools that make them look incompetent or irrelevant. So flashy UIs and buzzword-bingo rule the roost; actual analysis doesn't matter any more.

    Vic.

    1. ecofeco Silver badge

      Re: How rhetorical is this article?

      Modern Management™ uses statistics as a drunk uses a lamppost; more for support than illumination.

      ...and there you have it. Pure gold right there.

    2. FozzyBear

      Re: How rhetorical is this article?

      Couldn't agree more.

      The only other point that should be mentioned is how poorly the BI stack is normally implemented in the business Crap data in, crap results out

  11. Doctor Syntax Silver badge

    'The “average” user may well not understand statistics but that surely puts the responsibility onto the designer of the BI package to do the appropriate stats'

    No, if the "average"* user doesn't understand statistics they shouldn't be trying to use them. Business Intelligence - yet another oxymoron.

    *"Hacker was a very average minister"

  12. LSonne

    I actually think you ask for a lot

    The author claims this is simple stuff. I agree that statistic tests are important. The automated data manipulation would be great but, I think it is very difficult to do well unless you accept a very rigid data structure with clearly defined hierarchies.

    I am also not convinced BI's should look to much for patterns. It will take a lot of effort to make this work. In the meantime I can imagine piles of decisions based on spurious regressions.

    Well done analysis is hard and takes insight and some grasp with statistic methods. I agree that development could focus less on the shiny parts. Even then, I think we would be pretty far from having BI's that can automate this in a useful way. It's just not that easy.

  13. two_cents

    Any software that can do this?

    just wondering if anyone knows of any products that can do this even if partially?

    1. Gordon 10

      Re: Any software that can do this?

      Any stats package can knock this out in a dozen or so lines of code. (R, Matlab, Mathematica etc)

      Even Excel does a reasonably credible impression if you add in the analysis tool pack thats present but not loaded by default.

  14. Adrian 4
    WTF?

    nerd fail

    https://regmedia.co.uk/2016/04/29/nerd_fail_photo_via_shutterstock.jpg

    Really ?

  15. Uberseehandel

    Some of the better BI tools have an interface to R. Which may be used or abused to create representative data sets. Unless a BI tool has such a capability, or its equivalent, there are always going to be problems.

    The kind of people who authorise BI projects are unlike to know what the previous paragraph is all about. They should have the requirement explained, preferably by the practicing consultant statistician/mathematician that has been retained to validate what is being done. Few of us have done enough statistical work outside of university to be able to determine whether what is being done with the data is reasonable. I can recall learning a great deal about different statistical methods, but rather less about when to use which method and how to determine between two similar but different techniques.

  16. computinghomer

    How can you conclude anything from one set of numbers for sales. I would think a series of sales figures would be required for both male and female to make any conclusion.

  17. Anonymous Coward
    Anonymous Coward

    Business Intelligence?

    A few years ago I worked with a team of BI analysts. Sure, they did know statistics, on the whole they were intelligent and they were fairly comfortable with packages such as SAS, but "business".. err NO,! They were pretty clueless about the market and what the company was doing! On a number of occasions they presented the most idiotic "findings".

    As such I can see why the BI tool vendors try to target the business people - and visualization is an exciting and easy thing to offer, but frankly arming them with various statistic tests may be asking for trouble?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon