back to article ML fails: Loyalty prediction? Not really. And bonus prediction? Oh dear

You may never have heard of the NPS but if you have ever answered a question such as: "How likely are you to recommend our company/product/service to your friends and colleagues?" you have entered the Net Promoter Score zone. The Net Promoter Score is used to gauge the loyalty of a firm's customers and, since its introduction …

  1. James 51 Silver badge

    So those poor buggers on the phone or in the shops or street whom I tell I never buy something the first time I see it are buggered then.

    1. Jim Mitchell

      If you never buy anything, then you don't get the survey to fill out! You don't count. Sorry.

      1. James 51 Silver badge

        I do eventually but the sales person rarely has anything to do with my decision. I might go in for info but the decision is usually made somewhere else after some though.

  2. A Non e-mouse Silver badge

    I didn't used to mind taking part in these surveys for the account managers of our main suppliers. Then they slowly started leaning on me and said I needed to give them a 9 or 10 otherwise they were in trouble. Having read the article, I can understand why.

    I strongly dislike being told what answer to give for a question so I now decline to take part. My guess is that many of their other customers are taking the same line and they're getting fewer and fewer responses, making it harder and harder for them to achieve their target score. (Which in turn puts more and more people off and....)

    1. alain williams Silver badge

      No: you have not understood why

      Then they slowly started leaning on me and said I needed to give them a 9 or 10 otherwise they were in trouble. Having read the article, I can understand why.

      Whether the sales droid gets a bonus is not the real issue: once you start giving them a 9 or 10 then the marketing department will start shouting from its web site that ''90% of customers consider us excellent'', which will pull in more people who will then also be pressured to give artificially high scores.

      So: what is really going on here is that companies are using the social pressure (''that nice man who sold me a car wants to give his daughter a nice birthday present, what possible harm is there is me lying on this questionnaire ?'') to artificially raise the company approval score; this acts to the detriment of (potential) customers. The value of the commission paid to the salesman is trivial compared to the extra sales that are generated.

      Just another way of generating fake news.

      A consequence that the article authors did not understand.

      1. Anonymous Coward
        Anonymous Coward

        Re: No: you have not understood why

        It's not fake news, it's marketing.

        Marketing should be dealt with in the same way as unwanted dog poo.

        At first ignore it.

        If it continues, investigate further and confirm the source.

        If it still continues, apply fire to the source. A LOT of fire.

        Slowly, the world will become a better place.

  3. malle-herbert Silver badge
    Coat

    NPS...

    Should have called it NBS... No Bonus Suckers !

    Because in the end it's just a tool for the tools in charge to be able to deny bonusses for their employees while their bosses's bonusses are probably tied to different metrics...

  4. Lysenko

    Fails at basic logic...

    I wouldn't rate anyone at "10" for unless they were absolutely perfect in every way (which is impossible) and I wouldn't rate "1" unless they were a physical manifestation of pure evil. Neutrality is of course "5" as anyone who has ever seen a bell curve knows instinctively.

    1. Ultimate Evil

    2. Imbecility

    3. Incompetence

    4. Sincere effort

    5. Average

    6. Good

    7. Very good

    8. Excellence

    9. Preternatural

    10. Godlike

    1. David Knapman

      Re: Fails at basic logic...

      Neutral when the scale has 10 possible values means that you *ought* to treat 5 and 6 as equal. Otherwise, your neutral is 4 steps away from Evil but 5 steps away from Godlike.

      This is why if you _want_ to offer a neutral answer option you ought to have an odd number of values on your scale.

      (Note that the NPS system has a 0 which you've not assigned a value to, but apparently it's worse than Evil)

      1. Lysenko

        Re: Fails at basic logic...

        Zero? In that case, I can revise the scale by moving 1-4 down a place and creating a new "Adequate" rating at position 4 i.e:

        0. Ultimate Evil

        1. Imbecility

        2. Incompetence

        3. Sincere effort

        4. Adequate

        5. Average

        ... makes no difference to the frontline victims though because it just adds greater granularity of fail. Of course, I'm being honest here. If I actually get wind of someone using this kind of thing to scam specific employees out of their wages then I'll happily be completely dishonest in whatever way is most effective for gaming the system. I'm only honest when the rating is applicable to an organisation as a whole, such as habitually rating EE between 1-2 (on the scale above).

        1. Peter Ford

          Re: Fails at basic logic...

          Wow, EE get a 1-2?

          Are they really that much better than all the others?

          1. Lysenko

            Re: Fails at basic logic...

            EE can never reach 0 while Vodaphone exist, just as ES3 era JavaScript could never quite reach "worst language in the world" status because PHP has had a lock on that since it first emerged from Satan's backside.

      2. Rich 11 Silver badge

        Re: Fails at basic logic...

        (Note that the NPS system has a 0 which you've not assigned a value to, but apparently it's worse than Evil)

        0 = Lawyer?

    2. Simple Simon

      Re: Fails at basic logic...

      Yes. But.

      The last time you bought something online (eg. App, auction site, mail-order, hotel, etc) - did you go for the supplier with 3 stars out of 5, or 5 stars out of 5?

      As consumers, we should be satisfied with a supplier who has a score of 3 out of 5. But, we're not. We see that as a failure. As business owners, we have to ensure that our feedback score is always 5 out of 5 - or we lose business.

      So, we find that the scale has been re-calibrated. Maximum score doesn't mean "God Like". It now means "Everything went OK. The parcel arrived on time, and wasn't broken - and I received the item I paid for".

      1. find users who cut cat tail

        Re: Fails at basic logic...

        If 5 stars are a necessity, then they carry no information. Why bother faking it the complicated way? Do not actually allow any rating and simply display five stars...

        Also,

        https://xkcd.com/937/

  5. Anonymous Coward
    Anonymous Coward

    It's hard not to come to the conclusion.....

    ....that the authors of this piece have never worked for a large organisation with multiple layers of management.

    Otherwise it wouldn't/shouldn't have come as a surprise to them that some notional objective(ish) measurement tool has instead been transmogrified into a tool to support/extol managements objectives rather then measure their effectiveness.

    Did you really expect anything else?

    1. no-one in particular

      Re: It's hard not to come to the conclusion.....

      > Otherwise it wouldn't/shouldn't have come as a surprise to them...

      I didn't spot where the article expressed any suprise. Just a reasoned explanation of what is going wrong, so that we can all understand how it is going wrong, not just that it has been going wrong.

      Perhaps also a modicum of sadness that those who actually need to understand an article like this will be precisely those who would never read it - and it was ever thus.

  6. Brewster's Angle Grinder Silver badge

    "Since the probability of any one of your customers being a promoter is 0.7..."

    I probably haven't had enough coffee to fight through my cold, but I don't get that at all. If you model the customer as equivalent to rolling 1d10 then the chance of them being a promoter is 20% (a roll of 9 or 10); with a 10% chance of them being neutral and a 70% chance of them being a detractor.

    Alternatively, if the P/N/D distinction accurately captures human behaviour, you could argue there's a 33% chance of each outcome.

    Where the hell does 70% chance of being a promoter come from?

    1. David Knapman

      From the employer. Higher up in the article:

      Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D).

      Your employer has told you you can achieve your bonus if you hit these figures. The rest of the article is pointing out that it's untrue.

      1. David Nash Silver badge

        "Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D)."

        Maybe I don't understand statistics but how does that state that the chance of any one customer being a promoter is 70%? It merely states what you will get as a result of customers being a Promoter or not.

        1. Brewster's Angle Grinder Silver badge

          On reflection, and slightly less cold, I concede that the boss's target can be reused as a probability distribution. If the salesman's making that target exactly each month, and you pick a sale at random, then it will have those probabilities of being P, N or D. In effect the saleman acts as a convolution on the underlying probabilities.

        2. Anonymous Coward
          Anonymous Coward

          It's essentially ex-post vs ex-ante. If you take the employer's suggestion of how you can hit your target (ex-post) then they are the probabilities whereas ex-ante your original statement is true assuming a uniform distribution.

      2. Brewster's Angle Grinder Silver badge

        "Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D)."

        That's asserting you can achieve a bonus if the number of promoters are 0.7 * number_of_sales and the number of neutrals are 0.2 * number_of_sales. It has nothing to do with 0.7*0.7 or 0.7*0.2 which are the chance of two 70% events happening or the chance of a 70% and a 20% event happening.

        If you're right, and that's the reason they've used 0.7, then the stats is very wrong.

  7. I Am Spartacus
    Thumb Up

    Good article.

    Timely and helpful.

    1. Phil O'Sophical Silver badge
      Coat

      Re: Good article.

      So, 4/5 stars then?

    2. Anonymous Coward
      Anonymous Coward

      Re: Good article.

      It might have been sensible to point out that Net Promoter Score is actually a trade marked term, and the technique, although widely copied is actually a proprietary one.

      Other than that, the article is a fair assessment of the bungled and widespread mis-applications of the tool. At the individual the article is correct, but even at company level it is flawed, because there is another level of failure, and that is that it assumes that "not recommending" is a standard level of detractor, and that's rubbish. When a customer is really disaffected, it isn't that they "wouldn't recommend" a company, they actually become quite vocal telling people to avoid a company (like I do for Virginmedia). NPS completely fails to capture those real detractors. It likewise misses out the real advocates, who individually will score a company 9 or 10, but who are a small subset of those reported 9-10s.

      The real appeal of NPS is twofold: It gives a single number, so ideal for target setting and "performance measurement", and it is easily manipulated.

  8. Anonymous Coward
    Anonymous Coward

    They are useless

    I gave a scathing response to a carrier.

    I pointed out the management were incompetent, the call logging process was flawed, the first two weeks of so called service and continued denials were shockingly bad.

    After a few weeks (and threats to move thousands of pound of business elsewhere) did it finally get through to the real tech team. In 24 hours the issue was resolved.

    Guess who got the bad score?

    Yup the poor sods that actually fixed the issue.

    needless to say another scathing emails and phone call followed.

    1. Wensleydale Cheese

      Re: They are useless

      "Guess who got the bad score?

      Yup the poor sods that actually fixed the issue."

      This is a very real problem. There have been times when I would like to send a message of dissatifaction to megacorp for sending me through too many hoops to get to a techie who knows their stuff. I definitely don't want to adversely affect the techie, but I still want to get the message through that the support structure stinks.

  9. Joe Harrison

    sick of it

    In theory it sounds like a good idea to feed back your degree of satisfaction with a product or service. In practice I have had enough of filling in endless surveys every damn time I buy a pair of socks, especially if (as the article says) the questions can't be taken at face value and the answers are used to downvote someone's bonus.

  10. Nik 2
    Thumb Down

    Even the question is stupid

    How likely am I to recommend this company to a friend?

    0 - because I don't have any friends?

    0 - Because this isn't the sort of thing my friends and I talk about?

    0 - because I know my friends won't ever use this sort of company or service?

    Even when I'm really pleased with a company, it's probably less than a 1 in 5 chance that I'll tell anyone, for at least one of the reasons above.

    Hack me off enough and I'll go round reading every story on El Reg looking for an opportunity to mention how useless [redacted] are in the comments.

    1. Z80

      Re: Even the question is stupid

      This.

      The other day I called Barclaycard Merchant Services to find out how to perform some tests on a PDQ terminal that was intermittently failing to connect when processing transactions and I got put through to an automated survey at the end of the call asking how likely I was to recommend their services. To be fair they did ask during the call if i didn't mind that happening.

      Given my job role there's no scenario I can imagine where my view of payment processing providers would be sought or need to be offered. They honest answer would be 0 - it's never going to happen.

      I played along though and gave them a 6 because they guided me through the testing procedure efficiently (it worked this time) but offered no suggestions as to why it was sometimes failing.

      This was classed as dissatisfaction on their scale and they then wanted me to put into words why I felt this way. I suppose that's a way of obtaining more meaningful feedback but I wasn't prepared to offer any more brain power than that required to think of a number and press a button so I hung up.

    2. ma1010 Silver badge
      Holmes

      Re: Even the question is stupid

      EXACTLY! As a former non-PHB of mine said "If you do someone a really good job, if the subject HAPPENS to come up in conversation, they MIGHT mention that you did them a good job. If you do them a bad job, they will bring up the subject in every conversation they have with anyone and tell them you're a screw-up."

  11. BebopWeBop Silver badge
    Holmes

    Well, one obvious explanation why they carry on using it is that no matter what the sales record, they can remove the majority of salespeople from the bonus scheme?

  12. lukewarmdog

    doctors surgery

    My doctors texts me this after every visit.

    "How likely am I to recommend the surgery to friends and family"

    Vodafone have started randomly texting me too.

    "How likely am I to recommend Vodafone to friends and family"

    It's more a question of how likely am I to just delete the text upon receipt.. very likely.

    1. Anonymous Coward
      Thumb Down

      "How likely am I to recommend the surgery to friends and family"

      Not very likely because you keep hassling me with bloody texts. Feedback should be offered spontaneously by a customer (or equiv.) not relentlessly nagged for by the service supplier.

      1. Diogenes

        Re: "How likely am I to recommend the surgery to friends and family"

        not relentlessly nagged for by the service supplier.

        My record is 15, including 5 pieces of snail mail from the dealer about a car service earlier this year.

        This ended up annoying me so much I ended up giving them a score of 0 with the comment about the nagging. Got a phoned apology form the service manager, who then had the hide to ask me to rescore the actual service. I hung up. As they offered an extended-extended warranty if I serviced with them, and this is now expired, I will no longer use the dealer service.

        Just curious - how does a 'nil' response score ?

  13. This post has been deleted by its author

  14. Anonymous Coward
    Anonymous Coward

    Nothing new under the sun ...

    Frank Bettger had it all figured in the 1930s. He analysed the streets that bought most insurance, and allocated his time pro-rata. It didn't take long before he was getting 80% conversions from 20% effort.

    I mentioned this once at a sales pitch an NPS expert firm were giving to our board. But techies don't get consulted on spunking money to marketeers.

  15. Detective Emil

    Too bad …

    … Wetherspoons was already on my sh*t list, so I don't get that flicker of smug satisfaction for adding them on reading this. Well, except that I now know that, by not being a customer at all, I arguably cause less front-line damage than by patronising the outfit.

    1. Anonymous Coward
      Anonymous Coward

      Re: Too bad …

      Well, except that I now know that, by not being a customer at all, I arguably cause less front-line damage than by patronising the outfit.

      Don't you want to be a customer just so you can wear the Wetherspoons customer uniform? You know, a saggy, stained grey tracksuit, with bottoms having a knee level crotch so that as you waddle it looks like you've shat yourself? Fag in hand, three days of stubble, and ideally a proper ratboy face?

  16. allthecoolshortnamesweretaken

    I think I'm more of a disrupter than a detracter.

  17. Anonymous Coward
    Anonymous Coward

    Bad Targets?

    The company I work for used to have this kind of question on their customer portal and once every month or so if you logged in, it asked you how high you would rate it.

    There was a bug that caused me to get the survey every time I logged in (probably the way I was binning the cookies).

    I just answered 10 every time and when the followup question joyfully thanked me for such a high score and gave me the opportunity to fill in a free text box with the reason, I took great pleasure in typing "I work for the company and part of our bonus is based on the NPS. Bad targets drive bad behaviours!"

  18. DougS Silver badge

    I made a recommendation to a friend this past weekend

    I rarely do, but I became aware of a Chinese company that sells products (for a commercial kitchen, not IT related) for much lower prices than the competition. That alone wasn't enough to get me to recommend them, but the products are of very high quality as far as I can tell and I'm able to get service on them from a local supplier should it be necessary.

    The best part is that when I google the products everyone on the internet is selling them for the same price as that local supplier so I don't feel like I have to choose between a better deal that screws the local guy who I will need to count on for service or overpay to guarantee I have a happy supplier who will provide quick service if it is necessary.

    I still wouldn't feel like I could give them a 10, or even a 9, because I've only had a couple items for a few months - too early to gauge reliability. The price I paid was less than double what I'd pay for used "guaranteed for 90 days" equipment, and almost half of what I would have paid for new equipment from other companies so I felt it was worth taking a chance on a new player. If I'm still happy after a year or two then it's a 9, and if it is trouble free for 5+ years then I'll call it a 10. I guess I'm hard to please, but the NPS scores don't seem to jive with my recommendation threshold since they would consider 8 to be 'neutral'.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019