back to article Top VW exec blames car pollution cheatware scandal on 'a couple of software engineers'

Volkswagen America CEO Michael Horn has played the "rogue employee" card to explain how and why his cars' engine software cheated in pollution tests. While being grilled by the US House Committee on Energy and Commerce on Thursday, Horn said he only learned of the existence of the so-called emissions-hoodwinking "defeat device …

  1. Donald Becker

    I can readily believe that it was the actions of a few rogue employees. I've seen it happen.

    But if VW hasn't identified who those employees are this far along, they are incompetent or lying. This is a major incident that may cost billions of dollars. VW should have people working 16 hour days to figure out how this happened and how to mitigate the damage.

    1. Esskay

      There's also the fact that the VW Group CEO has already stepped down - you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up. This is Horn trying to save himself.

      1. bri

        @Esskay

        Au contraire, you do step down to appease shareholders and other relevant parties, when such a massive loss of value occurs. And I don't believe that Winterkorn wanted to clean up the mess created under his watch, with everyone questioning his every move.

        Horn is in full corporate damage control mode, as minimizing fallout is crucial to future survival of the company. I don't believe a word he said, but I fully understand why he said it. He also has no other choice. Unless he wants to endanger hundreds of thousands employees. The are no good choices for VW now.

        1. chivo243 Silver badge

          Re: @Esskay

          @bri

          "This was not a corporate decision from my point of view," Horn said while under oath.

          "I don't believe a word he said"

          Neither do I, but Horn believes it. Just like George Costanza said, "It's not a lie, if you believe it."

          Rogue employees? My ass, that's almost as bad as "some older boys made me do it." "A misinterpretation of a directive by "a" team in middle management are responsible." would be a much more plausible way to focus and defuse the guilt elsewhere.

          It will be interesting to see the testimony given in the US vs the testimony given in Germany. I get the feeling they won't be the same story.

          1. James Micallef Silver badge

            Re: @Esskay

            "This was not a corporate decision from my point of view,"

            Since the very moment this shitstorm broke I was expecting the moment that some 'rogue employees' were blamed. Predictable as night follows day.

        2. TheOtherHobbes

          Re: @Esskay

          >He also has no other choice.

          Let's see what happens when a few staff are fingered and they decide to leak emails from management.

          Of course it could have been a few rogue coders who just happened to write code that didn't need to be tested physically, and didn't produce any obvious unwanted side effects.

          Of course it could.

          Classy move, anyway. I hope he goes to jail.

        3. Uffish

          Re: @No good choices for VW

          The "all the fault of a very few engineers" argument was aired at the same time as the CEO left. If that is really the truth I will never buy another VW because you cannot trust the factory to make what it thinks it is making.

          VW has had enough time to find a solid chain of command from programmer to board level (all of whom, rather surprisingly, would be getting on in years and very fond of fishing). That scenario might not be the truth but it doesn't stink to high heaven like the current one. There are far better choices than choosing an indefensible defence.

        4. big_D Silver badge

          Re: @bri

          I agree, nobody would have confidence in Winterkorn, whether he had known or not. With a "clean" start, with a new CEO, there is less residue for people to point fingers at. (Sorry for the puns).

          What got me was this: Rep. Chris Collins (R-NY), himself a former engineer, noted that if this defeat device was the work of two Volkswagen engineers acting alone, the company's intellectual property department would have every reason to seek to patent the innovation unless they were directed otherwise by management.

          And if he knew anything about Europe, he would know that you couldn't patent that anyway. That is software, software isn't patentable.

      2. Charles Manning

        " you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up."

        Wasn't he due for retirement? If I was a CEO and had my retirement lined up, I'd retire to go fishing rather than face a shitstorm.

      3. Graham Marsden

        @Esskay

        > you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up

        And anyway, you have a nice golden lifeboat with a lucrative retirement plan set up...

      4. DButch

        The engineers involved would be in Germany, since the design was mainly done there working with Bosch, who supplied the pollution control system components. TDI diesels with the pollution control hack were shipped to a lot of countries, including Germany itself. It might not have required that much work on the part of VW engineers since it looks like it took advantage of (presumably) test modes in Bosch' own equipment - simply turned various features on (or off) to get the desired reduction in pollution during emissions tests.

        An article in the Telegraph states that Bosch sent a memo to VW in 2007 warning that use of the SW to defeat emissions test measurements would be illegal. Wonder if some VW engineer made an incautious comment in front of a Bosch representative leading to that warning.

        1. Mark 85

          Or, Bosch knew that VW was going to activate the test mode memo was merely a CYA memo.. The thing is, why wasn't the test mode deleted from the delivered code if it was "illegal"..?

          1. Alan Brown Silver badge

            "The thing is, why wasn't the test mode deleted from the delivered code if it was "illegal"..?"

            Because it's needed to stop the car panicing when it's on a rolling road.

      5. The Man Who Fell To Earth Silver badge
        Boffin

        @ Esskay

        What rock have you been living under? The software cheat originated in Germany, which is why it was applied WORLD WIDE. VW got caught in the US because US environmental requirements for autos are tougher than anywhere else as is testing.

    2. Lysenko

      I agree...

      ...there is a worrying lack of appreciation of how intense reality distortion fields can be and how much rationalisation can be manufactured to get past an: "I can't tell the Boss\Customers\Shareholders THAT! He \they won't like it!! <cringe>" culture.

      I can buy the idea that VW execs. didn't know about this because I can buy the idea that they created a culture where "positivity" was encouraged (promoted) and "negativity" was discouraged (fired) to the point that they were left bouncing around in an echo chamber of their own wishful thinking. I've seen it happen first hand. That sort of ignorance is still culpable of course.

      1. Ian 55

        Re: I agree...

        Robocop 2 had it covered perfectly:

        Minion: It'll be a feeding frenzy.

        Boss: It's only money. What about criminal proceedings?

        Minion: Well, we're looking at major indictments, sir. Prison terms.

        Boss: What about me?

        Minion: You know we'll do whatever we can.

        Boss: That's not good enough.

        Minion2: Sir? What if this was the work of one individual? A person who had her own agenda, wasn't in sync with the goals of our company?

        Boss: Well, that usually works.

        Minion2: A woman who was not a team player, who violated our trust.

        Boss: Well we'd need some evidence to support that.

        Minion: Sir, whether it exists or not, I know I can find it.

      2. I am not spartacus

        Re: I agree...

        "That sort of ignorance is still culpable of course.

        Exactly, although 'our engineers will do the impossible by next Wednesday' is worse.

    3. Paul_Murphy

      Other makers

      I wonder what the story at other manufacturers will be, since I doubt VW will be the only ones making use of technology to make their cars appear better to the buying public.

      If your competitor appears to be making cars that are so much better than yours, in terms of fuel efficiency at least, then what do you do?

      Get consigned to history or see what you can do to catch up??

      1. Gordon 10
        Mushroom

        Re: Other makers

        Or possibly be honest built it to spec and take the hit on your bottom line. There seems to have been a strong whiff of the profit motive about VW's actions - I suspect it was initially about building to a price for the American market - then it just grew by habit. I suspect they were trying to undercut BMW and Mercedes in the US market- whilst trying to also promote diesel as the fuel of choice as a differentiator. Up to now the VW TDI's have had a certain level of caché that in trying to capitalise on they have flushed down the toilet.

        I hope this "2 engineers" comment encourages one or more of them to turn states evidence and shaft the squirming CEO's good and proper.

        I find it intensely annoying that the default mode of any corporate drone these days is to lie, prevaricate and deflect instead of just telling the truth. There need to be much stronger laws on corporate perjury. i.e. both jail time for the Exec and % of world turnover fine for the company.

      2. IvoryT

        Re: Other makers

        Indeed, Lance Armstrong knew that feeling.

      3. Version 1.0 Silver badge

        Re: Other makers

        Every automobile manufacturer that I dealt with in the 80-90's would purchase a few of each of its competitors vehicle as soon as they hit the road and investigate many aspects of its performance. I find it very hard to believe that the VW trickery wasn't known within professional engineering circles in the engineering end of the business.

        Getting clean diesel engines to work is hard - it's unbelievable that other manufacturers didn't say, "Let's take a look at what VW are doing"

        1. TitterYeNot

          Re: Other makers

          "Getting clean diesel engines to work is hard - it's unbelievable that other manufacturers didn't say, "Let's take a look at what VW are doing""

          Indeed, modern diesel engines are much more expensive to develop than their petrol driven counterparts, so car manufacturers often cooperate in their design and development (I know of at least one British manufacturer that developed several diesel engines with a French competitor.)

          It does therefore seem a little unlikely that only VW know how to do this slight of hand with their engine emmissions. Whether others have put this know-how into practice as well, I guess only time (and screaming headlines) will tell.

        2. DButch

          Re: Other makers

          I saw a comment in one of the early articles after this broke that there were "a lot of questions" around how VW could get the claimed combination of power, fuel economy, AND low NOx pollution from a relatively "primitive" pollution control system. I suspect no one in the industry wanted to blow the whistle lest questions be raised about THEIR results.

          I've already heard another question about a somewhat less elaborate hack in a similar vein - control of urea injection designed to insure good results in test, but stop using it in regular driving to avoid drivers having to refill the urea solution too often (or at all).

        3. Skiper

          Re: Other makers

          "I find it very hard to believe that the VW trickery wasn't known within professional engineering circles in the engineering end of the business."

          That's when the conspiracy theory kicks in...

        4. I am not spartacus

          Re: Other makers

          "...purchase a few of each of its competitors vehicle as soon as they hit the road and investigate many aspects of its performance.

          Well, everyone does that, but the primary focus is what 'the others' do to keep the production costs down.

          However, if they think that Manufacturer A is doing a better job with, say, Diesel engines than they are, Manufacturer B will give increased attention to anything that manufacturer A is doing.

          That said, this is way more difficult than, say, 'Why are their seats cheaper than ours?'.

          But, yes, I would say that it is difficult to believe that at least some other manufacturers didn't have a very good idea that something odd was going on, even if they couldn't describe what the strategy was, exactly. And, how did the University researchers know to look at VW? Good luck, or did someone give them a word to the wise?

    4. BitDr

      Rogue employees...

      This smacks of a business decision made by marketing & sales; not a decision made by engineers.

      I have no doubt that the engineers came up with the implementation of the solution, in exactly the same way that I'm certain that the engineers were not the ones driving the bus that they would eventually be thrown under.

    5. Dr. Mouse

      I can readily believe that it was the actions of a few rogue employees. I've seen it happen.

      Really?! A significant innovation comes through, and no managers whatsoever know either who came up with the idea, nor sought details in order to patent it?!

      Yes, rogue engineers of all kinds sometimes slip something under the radar. But this wasn't slipping a back door in or a fudge to work around or hide an issue they couldn't fix, this was a blatant cheat which produced a clear competitive advantage over their rivals. If it had happened and noone in management had known, the engineers (or their dept, or their manager, or his manager etc.) would have had praise heaped on them for saving the company so much money and/or giving them such an edge over their competitors. They would also have immediately asked for the details so that they could patent it before their competitors could copy it.

      There is simply no way that knowledge of this could not extend at least a fair way up the management hierarchy. At the very least, there would have been a point where a manager asked about it and was told "it's better that you don't know...".

  2. Anonymous Coward
    Anonymous Coward

    In this instance, there's no way it was due solely to the actions of a few lowly engineers. It could only have worked at the scale it did with executive buy in.

    There is substantial documentation showing VW (at the highest levels) have been officially notified repeatedly for years about problems with their emissions. They've also previously been fined for rigging results, so this isn't just a one-off.

    It's unfortunate their execs feel lying is the correct path to take here. It's not unexpected though, as modern Western style management (US MBA driven) is all about short term thinking, corporate greed, and PR spin. :(

    1. Anonymous Coward
      Anonymous Coward

      So, even when it's Germans doing the lying, make sure everyone knows that the US is still the guilty party, eh? You sound like a Democrat blaming 7 years of Obama failure on the previous president.

      1. david 12 Silver badge

        You sound like a Repubican blaming 7 years of Congress failure on the President.

        1. Graham Marsden

          @david 12

          > You sound like a Repubican blaming 7 years of Congress failure on the President.

          ... when you've done everything in your power to block, damage or subvert everything that that President has tried to achieve.

          F(inshed)TFY

          1. Anonymous Coward
            Anonymous Coward

            Re: @ Graham Marsden

            > "... when you've done everything in your power to block, damage or subvert everything that that President has tried to achieve."

            Oh yeah, I forgot that the GOP was able to stop Obamacare. Not a single one of them voted for it! Oh wait, that didn't stop Obamacare.

            Well at least they stopped him from letting our military be hollowed out by neglect and PC inanity. Oh wait, they didn't stop that either.

            Well, there's all that new spending he wanted to pour into green energy...oh yeah, he got that as well. Too bad it all just diappeared down the rat hole.

            I could go on and on. Basically the GOP leadership has been Obama's BFF, which is why we now see a grass-roots revolt against those leaders.

            There is one thing the GOP did accomplish against Obama; The electoral results of the Obama Years has been utter devastation for the Democrats at all levels except the Presidency, and that won't last much longer either.

        2. Anonymous Coward
          Anonymous Coward

          > "You sound like a Repubican blaming 7 years of Congress failure on the President."

          Does that include his first three years when Obama's party dominated Congress? Otr the six years when they controlled the Senate? Pathetic.

      2. lighthabit

        what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?

        1. Anonymous Coward
          Anonymous Coward

          > "what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?"

          Both of them "fake but accurate," right?

          The unemployed class has expanded dramatically under Obama, but those are no longer counted as unemployed because they've given up looking for work. Still, those like you may still revel in a "low" unemployment figure, at least.

          About the deficit, you ought to google "us deficit timeline" if you want a bit of a shock. And let's not even talk about the obscene debt expansion Obama's party has laid onto future taxpayer's backs (when they controlled Congress and spending completely).

          But hey, all that massive new spending was at least used for vote buying thru Socialism and Eco-cronyism of the "shovel-ready" kind, not wasted on anything frivolous like defense. Who needs a military when Obama is such a great international statesman?

        2. Destroy All Monsters Silver badge
          Pint

          > what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?

          AWOOGA! AWOOGA! We have an Alternate Reality Intrusion! WHERE IS DOCTOR WHO?

      3. myhandler

        You don't sound like a dumb septic, you are a dumb septic. Chip on shoulder much?

      4. Queasy Rider

        And you sound like a Republican blaming previous administrations' failures on the present administration. Tit for tat.

    2. Destroy All Monsters Silver badge
      WTF?

      "From my point of view" ... WUH?

      Would these engineers 1) develop, 2) install 3) test, 4) QA test 5) run emission tests 6) write off roll-out?

      Maybe VW software development consists of three dudes behaving like a web development startup with a subversion repository and an "it creates HTML, ship it!" culture, but I don't think so.

      The idea of patenting an "defeat device" sounds daft though. First, is there anything to patent. Second, patenting it would mean a traceable item discoverable by anyone. "Procedure for quickly killing your wife using a bathtub". Nah, I don't think so.

      1. Anonymous Coward
        Anonymous Coward

        Re: "From my point of view" ... WUH?

        "The idea of patenting an "defeat device" sounds daft though."

        He perhaps did not express it very well but he is right. If management was told that a software change had fixed an intractable engineering problem, saving VW vast amounts of money, they would have wanted to patent the method.

        If Horn thinks he is telling the truth, then the problem we are seeing is that senior management doesn't understand how engineering works in their own company. That would be a first for Germany.

        1. Anonymous Coward
          Anonymous Coward

          Re: "From my point of view" ... WUH?

          perhaps, but they'll go that way if VW gets away with this. Plausible deniability. Hire a bunch of MBAs to run things who don't have an Engineering degree between them and they can claim anything at all.

        2. GrumpenKraut
          Unhappy

          Re: "From my point of view" ... WUH?

          > That would be a first for Germany.

          Sadly, not a first. Dunno about VW, but I have witnessed incredible technical ignorance at CXO level in tech-companies.

      2. Syd

        Re: "From my point of view" ... WUH?

        Pretty sure he wasn't suggesting they would patent the defeat device - they would patent whatever it was in the engine which made it run much cleaner than any of their competitors for the same performance.

      3. Dan 55 Silver badge

        Re: "From my point of view" ... WUH?

        I think he means VW group was the only manufacturer to get low diesel emissions without AdBlue. That should have been patented but it wasn't, something that with hindsight stinks from a mile away and not just because of the contamination.

  3. Anonymous Coward
    Anonymous Coward

    Even if it was down to "a couple of software engineers"...

    This would have to be attributable to unnecessary requirements, as if it was in the requirements it would be a project decision sanctioned by management.

    If there are unnecessary requirements, then the project does not comply with ISO 26262, which would also make the problem a management failure.

    The argument that it's down to rogue developers can only be supported if the software has been "hacked" without a proper development process being in place.

    1. Anonymous Coward
      Anonymous Coward

      Re: Even if it was down to "a couple of software engineers"...

      [No apologies for length. Share and enjoy.]

      It would be very surprising if the VAG incident comes down to "rogue engineers".

      Analysis of similar "engineering vs commercial pressure" precedents typically shows that occurences like this come down to "management culture" (not even "rogue management", just the routine way an organisation does its business).

      Even if an individual engineer does something stupid, the *organisation* is supposed to have a set of processes (and in some cases even "corporate ethics") to stop the stupidness reaching the outside world. Realistic system testing, proper QA processes, and such like tend to be helpful but have an unfortunate tendency to be seen by HQ as "non value added".

      When the company loses tens of billions of dollars because such "NVA" tasks are skipped, maybe it's time for HQ to rethink that viewpoint.

      I mention "corporate ethics" because large companies I am familiar with claim to have formal corporate ethics programmes, whistleblower hotlines, and such. I'm afraid that experience in many organisations (private and public sector) suggests that these ethics hotlines etc exist for cosmetic reasons only.

      One organisation I knew had such an "ethics programme" but there was also an informal but widely understood and very clear "don't rock the boat, or else" policy: senior engineers/managers do not make mistakes or misjudgements. To even think aloud about the possibility, when a well-informed independent outsider would likely be asking "why is this approach a good idea, show me the documented arguments in its favour", would be a career-limiting move.

      Below is a very brief extract from a presentation given to an Oil and Gas Industry conference on some cultural aspects of the 'modern' large-organisation engineering process. The presenter is not an engineer but a man with a history of investigating how disasters happened. Do you think his thoughts might be relevant in this picture?

      SEVEN THEMES OF NIMROD

      11. The following seven themes struck me forcibly as I began to investigate the Nimrod story:

      (1) Complexity. The Byzantine complexity of the organisation, the rules, the standards, the safety processes in the MOD was remarkable. Complexity and change had become the altar at which a lot of senior people worshipped – but had become the problem rather than solution.

      (2) Dilution. The immediate casualty of this complexity was a dilution of responsibility and accountability – it was difficult to divine who was responsible or accountable for what. Accountability is the ‘reciprocal’ of Responsibility.

      (3) Management by committee and consensus. There were more committees, sub-committees, working parties etc dealing with safety-related matters than the UN.

      (4) Lack of challenge. There was a culture which rewarded conformity rather than the asking of awkward questions.

      (5) Migration. There was a migration of decision-making and budgetary power away from those with most direct working knowledge to those sitting in warm offices back home.

      (6) Triumph of generalists over specialists. There was too little appreciation of engineering specialist skills, too great a reverence to for the young MBA.

      (7) Paper safety. Safety was increasingly a paper, coloured diagram and PowerPoints exercise, rather than a people, process and cultural matter.

      From Charles Haddon-Cave's presentation on "LEADERSHIP & CULTURE, PRINCIPLES & PROFESSIONALISM, SIMPLICITY & SAFETY" at a 2013 conference marking the 25th anniversary of the Piper Alpha oil rig disaster.

      https://www.judiciary.gov.uk/wp-content/uploads/JCO/Documents/Speeches/ch-c-speech-piper25-190613.pdf

  4. oldtaku Silver badge
    Thumb Down

    Just when you think they're in deep enough already they lie their asses off again. The whole company really has a sleazebag culture, apparently.

    1. Adam 1

      There are Lies coming even from their board.

    2. DropBear
  5. Number6

    Software Code Reviews?

    It doesn't say much for their software management processes. If you've got a couple of engineers with an idea, they have to spend time developing the algorithm to recognise and defeat the test, then check it out by seeing if it's capable of recognising test conditions reliably enough to be useful. All this takes time and resources, which presumably the management didn't notice?

    Then you have the formal code review before you release production code. Generally this involves letting other people look at your source code, may require you to have all your curly brackets in the correct place and use the correct combination of tabs and spaces and possibly even go through the functionality of the code and check out the results of your regression tests. Then you get the reviewers to sign off that it's OK and does what is required. All without them noticing the big block of source, the function of which wasn't clearly explained.

    Yes, clearly the work of a couple of rogue software engineers who are capable of amazing feats when it comes to hoodwinking the system.

    1. Voland's right hand Silver badge

      Re: Software Code Reviews?

      My exact thought. If this is a reflection of their software management process all of their cars MUST BE TAKEN OFF THE ROADS IMMEDIATELY.

      After all, VW is the most "software driven" mass production car on the market. Even the fecking lights are driven of individuial CANBus controller. So if their software management process allows "a couple of engineers" to introduce bespoke engine management code that alters the engine spec, then who the hell knows what lurks in the steering, airbag or breaks controllers.

      So if their QA process _CAN_ allow that, then all of their certifications ever since they moved to total CANBus-ing must be immediately withdrawn until a _PROPER_ review is complete.

      1. Destroy All Monsters Silver badge
        Go

        Re: Software Code Reviews?

        MUST BE TAKEN OFF THE ROADS IMMEDIATELY.

        It would only be a just reward for dumb shit coming out of management mouths.

    2. Anonymous Coward
      Anonymous Coward

      Re: Software Code Reviews?

      And don't forget the testing regime. S/W does not into production vehicles without testing. There would have been tests to determine whether the "defeat device" s/w properly detected testing and lowered emissions. (A sad part is that no whistel-blowers came forward. The Economist claims tthat this was an open secret amongst the community.)

    3. Anonymous Coward
      Anonymous Coward

      Re: recognise and defeat the test

      My understanding is that the fuel system's baseline behaviour will pass the test, and "normal driving conditions" which include things like steering and suspension movement are what trigger the behaviour that improves performance while increasing emissions. So there isn't a rogue subroutine somewhere, checked into RCS by a rogue developer, that runs in the background looking for test conditions.

  6. Big Ed

    Somebody In Management Had To Know

    So VW Engineers create a design, and they know the levers to push and pull to affect mileage and pollution levels. They have phenominal testing labs and reams of test data. When they create a car design, they would estimate the expected mileage and pollution levels. When the car rolls into testing and the results come in so good; someone should have been comparing the actual results to estimated results. And given the variance, some engineering bean counter has got to be asking why the results are so good; if nothing more than to bottle it up for the next design. And then the great achievement makes its way up the management chain.

    And nobody says great achievement and passes out bonuses and promotions for exceeding estimates?

    So sorry Mr. VW CEO; your story of two rouge software engineers doesn't pass the stink test.

    Me thinks the FBI needs to be brought in to find the scoundrels who got the bonuses and promotions. The evidence is all there in project files and in the reviews in personnel files if someone just looks.

    The US Govt was defrauded with lying test results, and more importantly consumers who bought the cheating cars got screwed.

    1. elDog

      Re: Somebody In Management Had To Know

      Calm down. There were some mistakes made.

      We'll happily take our 8Bn U$ as a fine since we made 20Bn U$ in sales we otherwise wouldn't have made.

      And since we're part of the top echelon CEOs/appartchiks of industry, we don't expect to have to change our vacation schedules. After all, "we're too big to fail!"

      1. zb

        Re: Somebody In Management Had To Know

        "We'll happily take our 8Bn U$ as a fine since we made 20Bn U$ in sales we otherwise wouldn't have made"

        That's what Ford thought when they did a cost/benefit analysis and decided that it would be cheaper to kill a few people than fix the design fault in the exploding Pinto. They were wrong.

        https://users.wfu.edu/palmitar/Law&Valuation/Papers/1999/Leggett-pinto.html

        Can you imagine an American jury deciding on punitive damages against a foreign company fraudulently selling a product (diesel engined cars) that American manufacturers don't make? How many zeros would you like? BP are handing over 50 billion to avoid the jury lottery and legal costs.

        Time will tell but I very much doubt if VW are forced to pay at least the last ten years' profits earned in the US.

    2. Anonymous Coward
      Anonymous Coward

      Re: Somebody In Management Had To Know

      > "...more importantly consumers who bought the cheating cars got screwed."

      No, they got better performing cars. Okay, those cars emitted a bit more NOX, but hey, everyone gets screwed by that, not just those car owners.

      1. Big Ed

        Re: Somebody In Management Had To Know

        Big John.

        Current owners are screwed because the resale value of their cars has dropped. And they will get screwed again when the software upgrade gets applied and their cost of operation goes up because thier gas mileage has dropped.

        And you can be sure that the state emmisions testors are going to fail their cars until the owners get the software patch applied.

    3. Commswonk
      Happy

      Re: Somebody In Management Had To Know

      "...two rouge software engineers...

      They should be easy to spot then. Or perhaps not, given that there ought to be "red faces all round"...

      1. Ralph B

        Re: Somebody In Management Had To Know

        I think the "rouge software engineers" may be red herrings.

        1. Anonymous Coward
          Anonymous Coward

          Re: Somebody In Management Had To Know

          Possible TERRORISTS, ancient members of the "Rote Armee Fraktion".

        2. Doctor Syntax Silver badge

          Re: Somebody In Management Had To Know

          'I think the "rouge software engineers" may be red herrings.'

          They were caught red handed.

  7. sysconfig

    I'm furious

    I bought a relatively recent A6 TDI not too long ago, and sure enough it is affected, according to their VIN check. All that check spits out is that Audi will eventually get in touch.

    Meanwhile, the CEO in Germany steps down - right thing to do, but he should have gone without severance package. German prosecutors raid offices and private properties of employees.

    The CEO in the US lies his arse off; and yet again somebody down the food chain is blamed?

    I want to see people very high up arrested and charged for this shit. It's very hard to believe that management had no idea. Of course they knew. On top of that, the company has been warned by Bosch (one of the main suppliers) and their own employee many years ago, as reported on El Reg: http://www.theregister.co.uk/2015/09/27/vw_emissions_scandal_car_maker_warned_cheatware_years_ago/

    What the f*** is going on? For me to restore trust in the brand and have me buy another Audi in the future, heads must roll, and owners must be compensated, because from my understanding the recall will have one or more of these effects:

    - emissions and therefore tax goes up eventually

    - performance goes down

    - mpg might change

    - at least several hours of my time will be wasted when its fitted

    - resale value of the car will drop (brand trust issues and/or changed performance)

    This is nothing short of organised crime.

    1. Doctor Syntax Silver badge

      Re: I'm furious

      "the CEO in Germany steps down - right thing to do"

      That's it, it was the right thing to do.

      The head's taking the blame and resigning, whether or not he had been directly involved is an old tradition. It looks like Winterkorn knew that and he deserves respect for honouring it.

      It makes a good deal of sense for the organisation as a whole and not just because he takes a good deal of the opprobrium with him. Taking responsibility is one of the things top management is paid for. If the senior management knows their jobs will be on the line they will not foster a culture of plausible deniability, they will want potentially serious issues passed up the line where they will then have a chance to quash them. The organisation becomes better managed as a result.

      Horn seems to be trying to push responsibility down the line. It's not a good idea. He's undone some of the good that Winterkorn did by walking, he's fostering a culture of plausible deniability and he wasn't even believed.

  8. Malcolm Weir Silver badge

    It's clear that the development of the "cheat device" was not the work of rogue engineers; as @sysconfig notes, the people who did it sent letters, etc. So the that shifts the theory onto much firmer ground: who approved which features should be in embedded in which release? And now you have a potential conversation that goes a bit like this:

    Engineer (manager): "We've been working on the (configuration of the) emissions control system, and have got much peppier performance/better mileage/cuter boothbabes".

    Business unit executive: "Will it pass the emissions tests?"

    Engineer (manager): "Yup".

    Granted, there's a certain amount of "Thomas a Beckett" going on ("Who will rid me of this turbulent emissions control regime?"), but I don't find it entirely implausible that some manager-of-engineering instructing a minion to leave the "emission test detection" code enabled and NOT reporting that up the chain of command, because he likely would have been engaged in a CYA over the fact that the bloody motor wasn't performing to plan.

    1. annodomini2

      Detection of the Running on a dyno is essential, certain features need to be turned off, traction control/stability control being the main ones.

      Front wheel drive car, front wheels driving, rear wheels going nowhere, traction control will think the wheels are spinning and cut power.

      Obviously, this is a simplistic view and depends how the algorithm is implemented, but the principle is there.

  9. a_yank_lurker

    Fall Guys

    Rogue coders did this which means VW has no software management. Where are the specs, the unit tests, code review, final tests, and signoffs? These should exist somewhere. Also, it is quite likely the coders, reviewers, and testers were likely unaware of the regulatory requirements and how they were translated into a specification.

    1. DainB Bronze badge

      Re: Fall Guys

      Calm down, software cam from Bosch who developed controller, they also warned VW that this mode should never be used in production cars. Why it was there ? Who knows, but it was VW choice activate it or not.

      1. Anonymous Coward
        Anonymous Coward

        Re: Fall Guys

        It was there as it is necessary for the vehicle to know when it's being tested. Otherwise putting it on a dynomometer is going to totally freak it out with contradictory inputs. Some telling it's moving, others not. Now determining it's being tested and putting it in an special exhaust regime is new. I seriously can say some bright boy got a pat on the back. However, now, they are looking for the phone number for Assassins-R-Us in their CEO/Chairman edition of the phone book.

        1. DButch

          Re: Fall Guys

          I could also see (well, speculate) that some non-standard modes might be useful when attempting to run diagnostics on the emissions control system itself for repair. I've certainly put a LOT of special test code in some of my SW designed to inject errors that are otherwise hard to trigger so I can test obscure error paths and make sure some of the people implementing and testing downstream code get a nasty shock if they trust my parameters too much...

          Before anyone protests, it's all under a strictly defined "test mode" compilation flag AND per management directives, of course.

      2. JeffyPoooh
        Pint

        Re: Fall Guys

        DainB "...software came from Bosch..."

        Some/most of it.

        I'd be surprised if the 'defeat device' section was theirs.

        If VW has coders (yes), then them.

        1. This post has been deleted by its author

  10. John Tserkezis

    Would be interesting to see if a text search of the source reveals that "Michael Horn" appear right next the critial code that changes the maps in that oh so special way.

    1. Mark 85

      I sort of doubt that you'll find his or any high level execs signature or name on a sign-off sheet or anywhere else. Now it's possible that those "rogue engineers" might have made some interesting comments in the code.

      1. a_yank_lurker

        I am think the post means the source code has a dated comment stating the relevant sections were initiated by management decree with the name of the PHB who ordered it.

  11. frank ly

    Interesting

    "You're telling me these engineers snuck that code into the software and no one said this is breakthrough technology, we need to patent this," said an incredulous Collins.

    He does have a point that management (and many other people) would have been incredibly incurious as to how the cars suddenly started showing much better performance than before the software was modified.

  12. DanielN

    There is no need to speculate about who did what and why. In due course tens of thousands of emails will be subpoenaed and hundreds of people will be questioned under whatever the German version of penalty of perjury is. VW will have their collective nose well and truly rubbed in whatever they did.

    As for repairs, the least polluting would be to switch the fuel from hydrocarbons to ammonia. Sure, you cannot buy it anywhere and it is toxic as hell, but the NOx problem will be GONE. ;-)

  13. Yugguy

    Let the blamestorm begin

    As management cover their arses.

    Management steal the credit when workers do well and heap the blame on them when things go wrong.

    And the higher up you get the cuntier you become.

  14. Ilgaz

    Does he remember Monica Lewinsky scandal?

    An American president almost lost his job and went to prison because he lied about his relationship with Monica.

    Let's hope they have a single line of code/checkout/comment/log to prove that claim.

    While on it, do you still think rms is a fanatic?

    1. Anonymous Coward
      Anonymous Coward

      Re: Does he remember Monica Lewinsky scandal?

      "An American president almost lost his job and went to prison because he lied about his relationship with Monica."

      But that was serious, unlike shafting your suppliers, customers and the government, which is just business.

    2. DButch

      Re: Does he remember Monica Lewinsky scandal?

      No, he came nowhere near losing his job OR going to prison. He was impeached, and the trial in the Senate collapsed because he had NOT lied in court by the definitions the prosecution agreed to use, they couldn't find anything else to accuse him of, and a fair number of Republicans turned out to be having their own affairs.

      This really drove the Republican Party insane because he proved that:

      1) He was a lot smarter than them and their lawyers and

      2) Republicans don't get much sex - and certainly no oral

      Even if he'd been convicted, removal from office would have been the only penalty.

  15. Lostintranslation

    Not me guv.

    It's the Google Streetcar Alibi.

    1. Destroy All Monsters Silver badge
      Paris Hilton

      Re: Not me guv.

      I don't understand?

      1. Ian 62

        Re: Not me guv.

        The wifi slurp when street car was doing the rounds. They hoovered up data 'by mistake', a rogue engineer was blamed.

  16. Anonymous Coward
    Anonymous Coward

    They knew all right …

    I have been told that VW management not only knew what was happening, but were also preparing for the eventual fallout by setting up an internal fund to pay for the fines. They blame the Government for making the requirements to difficult.

    … I’ve been told.

    1. JeffyPoooh
      Pint

      Re: They knew all right …

      AC "... making the requirements TOO difficult."

      It's likely that the more complex urea injection systems actually do meet the test standards without cheating.

      This doesn't mean that the test standards reflect real world conditions, a separate topic (but related).

  17. Anonymous Coward
    Anonymous Coward

    Anyone else think that the reason Michael Horn was the person selected because He didn't know about it, or not enough anyway, and had management's word that they "...knew Nothing. Nothing!" That he won't have to worry about a perjury conviction?

    Actually having VAG senior management on US soil anywhere probably isn't a real good idea right now, unless all the right donations were made in the past and penciled in for future campaigns.

    1. Anonymous Coward
      Anonymous Coward

      ...Michael Horn selected because...

      He should have dressed up as Sargent Schultz of Hogan's Heroes fame.

  18. David Roberts
    Holmes

    On the other hand....

    .....the small piece of code which set the values for they key parameters in the software to set up the emissions profile was configured by two very pissed off programmers kept behind over a public holiday because a management cock up had made the project run late.

    As is often the case they were the only technical people involved in the project beneath 15 layers of increasingly clueless management, most of whom were from Marketing.

    So they wrote an obfuscated regular expression to set the values. When the code was viewed they said "It's just simple maths - obvious, really." Nobody wanted to admit they didn't understand simple maths so it passed the limited review and went straight into production to meet the project timescales.

    For the first few weeks the programmers waited for someone to challenge them because it was such an obvious "Fuck you!" ploy but nobody mentioned it and they were redeployed onto other mismanaged projects and since they didn't care about the company and the management they eventually drifted off into the mists of wherever pissed off programmers go, and were lost to history.

    Yep - I could believe that.

    1. Francis Vaughan

      Re: On the other hand....

      Nice idea but zero chance of being the case.

      ECU code is some of the most reviewed and tested code on the planet. It tends to make Space Shuttle code processes look weak. Somewhere there was code that took in steering wheel angle measurements and affected the engine exhaust recirculation control actuators. Yet the guys that write the code in the ECU don't write the specifications, that comes from the guys that design the engine, and work out the combustion mechanics and engine maps and algorithms. Those specifications are worked over with a fine toothed comb. This is a hard real time system, the entire system will be specified to hard real time deadlines, and the software artefact understood not just to the machine instruction, but down to the clock cycle. There are many engineers involved in this. Something as utterly weird and blatant as steering affecting the EGR will never get past the levels of design and review that are needed in an ECU. You might manage to get the needed bits into the design and coded, but it would require complicity of a range of engineers and their managers. And it would require continued vigilance to keep the lid on things.

      One can imagine the idea coming from one of the software levels, and being percolated up the chin just high enough for someone middling senior to give development of the hack the green light. The hack may have languished unto pressure came from way up high that a fix was needed from the engine division or there were serious problems with sales to come. Then the hack may have obtained a life of its own and got into the product, possibly with some higher managers simply realising that it would be better not to ask how things were fixed.

      1. John Sawyer

        Re: On the other hand....

        @Francis Vaughan: Yup, the exact details matter in determining whether this was done by "rogue engineers" or something bigger. You can't have code in an ECU that does something as bizarre as shutting off parts of the emissions controls depending on steering, without a lot more than even two of the high-level engineers knowing about it.

      2. Graham Cobb Silver badge

        Re: On the other hand....

        I think your view of the ECU code is much too naive. There is no piece of code that says 'if (steering_wheel_change < 5) defeat_testing_mode();'

        Presumably what happened is something like... The ECU parameters were originally set up to meet the requirements. Then, during testing, someone discovered the engine was really sluggish during "high performance" manoeuvres (acceleration to overtake a lorry on a single carriageway road, for example) so a "high performance" ECU parameter mode was added, just for really short high performance needs, and so it could ignore medium-term issues like emission controls. I presume every engine does that -- the specs and tests are averages over time, not requirements that instantaneous values are never exceeded under any circumstance.

        After a while, the 'high performance mode" got enabled more and more (or more variants of it got introduced), in order to improve real-life performance during road testing. All for specific short-term issues, so emission controls didn't matter. Eventually, it worked out that the engine was never actually running with the original parameters except during testing.

        Of course, that doesn't mean that people didn't eventually realise what had happened, and that the effect was that in real life the car always had worse emissions than were shown in testing. But, by then. no one had the guts to stand up and say "we can't ship this -- we need to go back to the drawing board" -- maybe the future of a whole engine factory (employing everyone in a town) was tied to the success of that engine. It would have been easy to rationalise for oneself that "the out-of-spec ECU parameters were all for specific driving styles, so it is the user's driving style which is out-of-spec, not the engine", or "the California specs are truly unreasonable, so as long as we are minimising emissions, that is really OK".

        People can, and must, be held responsible for allowing the resulting engine to ship. But I will not be surprised if there was no deliberate intention in advance.

    2. Destroy All Monsters Silver badge
      Trollface

      On the other hand.... the hand of Lucas!

      For the first few weeks the programmers waited for someone to challenge them because it was such an obvious "Fuck you!" ploy but nobody mentioned it

      VW Code -- The Phantom Menace!

  19. Anonymous Coward
    Anonymous Coward

    Doesn't matter

    Just like Rebecca Brokes (in the phone hacking thing), if you take the exec salary then you are responsible, no question.

    You have rogue software engineers, your problem, you are C level, that means responsibility for more than just getting a big pay cheque.

    1. Anonymous Coward
      Anonymous Coward

      Re: Doesn't matter

      "Just like Rebecca Brokes (in the phone hacking thing), if you take the exec salary then you are responsible, no question."

      Come now, she was completely vindicated, innocent of all charges and is back in the job again.

      And Carly Fiorina is trying to get the Republican nomination.

      I think you may have missed a point about how big companies work.

      1. Peter Simpson 1
        Happy

        Re: Doesn't matter

        I think you may have missed a point about how big companies work.

        A former boss referred to it as "f*ck up and move up"

  20. Anonymous Coward
    Anonymous Coward

    Usual CEO behaviour. No excuse.

    The idiocy and arrogance of Horn's position is astonishing. If you are the CEO of a company then it is your duty and job to know every major strategic decision, to guide the company in implementation and to uphold and support the direction and approach, always being mindful of the need to adapt, change and react. If you can't do this well (and it's a tough role), then you shouldn't be a CEO.

    So: either he knew about VW's major decision to 'adapt' their engines' emissions profile under test conditions, therefore he's not fit for purpose, as it was a truly bad decision he helped support; or he didn't know at all in which case he's not fit for purpose as he had no idea about a major decision taken and therefore what his company was doing. The same goes for the entire directed graph (it sure as hell won't be a chain) of command spreading out from the supposed 'rogue engineers'. Either they knew and should've spoken up, or they didn't know but should've known.

    The sad thing is that in truth the people who wrote the code will be punished the most but are probably the last ones to blame (they are still culpable). It's very likely they were stuck in a culture that demanded positive results in impossible situations - and that can only lead to smart people making dumb decisions.

  21. The Vociferous Time Waster

    Source code review

    Somebody hit commit - find that somebody

    1. Anonymous Coward
      Anonymous Coward

      Re: Source code review

      "Somebody hit commit - find that somebody"

      More years ago than I care to remember I turned down a job at a company designing ECUs owing to the requirement of frequently spending a week at a time in the cab of an artic going around Europe monitoring engine performance against software revisions. (Sorry guys, but the alternative was the occasional business class one week trip to the US, difficult decision).

      Between commit in the R&D lab and the software hitting a road vehicle in a showroom is going to be a lot more testing and signoff by people with laptops and telemetry. That's going to go for the emissions test too.

      Either there is a whole chain of signoffs or we would have to assume that VW is incompetent. Either case doesn't look good for the share price.

  22. Anonymous Coward
    Anonymous Coward

    I say we take him at face value...

    The CEO of VW has stood up under oath and said that it is possible for rogue software engineers at VW to inject arbitrary code in to the engine management software of your car with no oversight or questions raised. Additionally, he has implied that VW software is not well version controlled (otherwise we'd be done with this investigation), that systems requirements are not captured and checked against before code is added to the project (otherwise the rogue contribution would have been spotted), that there is no upwards verification of compliance to design/implementation/test standards (otherwise this would have been spotted by the review chain), and that corporate culture in the engineering teams encourages not questioning the work of your colleagues/reports/superiors (otherwise a non-rogue employee looking at the code base or the performance of the engine would have noticed).

    That should terrify anyone running anything built by VW. If popular press was sharp on ISO 26262 and common software development and project management techniques, this would become the bigger and more damaging story. The half-hearted attempt to throw employees under the bus is as good as saying they have no idea what is going on in their safety critical systems. All of them. Globally. Hopefully someone will think to pull on this thread and see how far it can go.

    1. DropBear
      Trollface

      Re: I say we take him at face value...

      Well of course we did have impeccable versioning but you see m'lud the leopard kicked over the filing cabinet with all the whirring things inside and cor blimey, everything got a bit jumbled up so we can't really tell who dunnit no more, honest, guv!

      1. Dr. Mouse

        Re: I say we take him at face value...

        That should terrify anyone running anything built by VW. If popular press was sharp on ISO 26262 and common software development and project management techniques, this would become the bigger and more damaging story. The half-hearted attempt to throw employees under the bus is as good as saying they have no idea what is going on in their safety critical systems. All of them. Globally. Hopefully someone will think to pull on this thread and see how far it can go.

        Exactly!

        I used to work for a company which build semiconductors, some of which were automotive grade. Even for such minor components (think in the range from diodes and transistors to switching regulators, but nothing more complex) the complexities of meeting the ISO requirements were high. Everything, from design to testing code to test result storage, needed to be controlled and audited.

        If VW are saying any part of their ECU software could have arbitrary code injected by a rogue employee, they are admitting they have no idea what code is in the ECU. How do we know that terrorists haven't infiltrated the company and programmed all the cars' throttles to open wide at a particular time? This would be a very effective terrorist plot, causing vast destruction world wide (think of how many VW diesels are on the motorway at any time!)...

  23. Anonymous Coward
    Trollface

    Rogues!

    Just thinking outside the box here. Could those rogue software bods have been recruited to develop the Windows 10 privacy code?

  24. Joe Cooper

    No way

    This is not even a clever lie.

    For this to be true – that only software engineers colluded to make this happen – is to say the software engineers were the _only_ ones to predict the problem. It's to say that experienced mechanical and chemical engineers built an engine, were unaware of the chemical reactions that went on inside it, and failed to notice when the software people caught _and masked_ their error.

  25. sabroni Silver badge

    it doesn't matter if the management knew.

    They're paid to be responsible. They are responsible.

    1. Wommit

      Re: it doesn't matter if the management knew.

      @Sabroni, you don't seem to know much about modern MBA business practices. This is _VERY_ much the responsibility dodging behavior that I expect from large companies. I'm actually surprised that Winterkorn resigned rather than "staying to give the company stability and guidance through this difficult period."

      1. Number6

        Re: it doesn't matter if the management knew.

        Winterkorn probably decided he'd live longer and have a more enjoyable life if he quit. He might not have known how deep the shit was, or indeed that it existed, but having been dropped into it and discovered the hidden depths, he decided to get out pronto.

        As for execs and mistakes, par for the course. Screw up one company, there's always another directorship on the horizon unless you screw up really badly. Even some of those have come back after a bit of time inside.

  26. Will 28

    So we can safely conclude...

    ...that there might be other pieces of unauthorised code lurking in VW's engine management systems. In fact if it's possible for engineers to sneak in code to so fundamentally affect the workings of the engine without anyone noticing, it doesn't seem possible to deny that someone could potentially have placed code to make the engine explode on a particular date.

    To my mind he appears to have said that as a matter of security all VW cars with an engine management system (which I assume is just about all VW cars), must be immediately recalled.

  27. amanfromarse

    See this article for a completely different slant on

    http://www.wsj.com/articles/vw-emissions-probe-zeroes-in-on-two-engineers-1444011602

    Why Horn made himself look like a lying moron by saying '2 rogue engineers' when he was referring to the heads of R&D departments is inexplicable. Or perhaps it says something about how he views the VW hierarchy.

    1. Destroy All Monsters Silver badge
      Paris Hilton

      Re: See this article for a completely different slant on

      The two men, Ulrich Hackenberg, Audi ’s chief engineer, and Wolfgang Hatz, developer of Porsche’s Formula One and Le Mans racing engines, were among the engineers suspended in the investigation of the emissions cheating scandal...

      The details haven’t been made public, but Volkswagen’s investigation is focused on Messrs. Hackenberg and Hatz, Mr. Winterkorn’s top aides during his tenure at Audi, as well as Heinz-Jakob Neusser, head of development at the VW brand, people familiar with the matter said.

      Well yes. Basically three members of the board, then?

      Mr. Winterkorn became chief executive and put Messrs. Hackenberg and Hatz in charge of the Volkswagen R&D group and engine development. The two engineers had worked together devising new Volkswagen cars during the critical years when the vehicle maker struggled to develop diesel engines to conform to tough U.S. nitrogen oxide emissions standards.

      Months later, in August 2007, Volkswagen canceled the licensing deal for BlueTec with Daimler because it didn’t want to use its competitor’s brand. It rebranded the company’s diesel engine TDI—for turbocharged direct injection.

      I see...

      1. John Sawyer

        Re: See this article for a completely different slant on

        Hackenberg??

      2. Peter Simpson 1
        Happy

        Re: See this article for a completely different slant on

        So, a few rogue engineers...who just happen to report to Mr. Winterkorn, and are in charge of engine development and R&D. And Winterhorn knew nothing about it. Yup, just a couple of low level rogue engineers...you just can't keep an eye on all of them, and this sort of thing DOES happen every now and then.

        Was that a flying pig that just went by? And I think I just saw the Devil on skis...

  28. Camilla Smythe

    Did they miss a trick?

    <Flight of Fantasy Mode>

    Let's say that there are a number of measurable parameters and a number of controllable parameters within the Engine Management System which in combination affect performance and NOx emissions.

    Now, if I were to be presented with such a problem I would be inclined to try writing the software and its associated algorithms to goal seek a performance/emissions optimum for any particular set of operating circumstances.

    It may be the case that under certain conditions the solution fails specific test requirements but overall the solution averages a performance over the time of a particular driving cycle that meets or exceeds the requirements.

    As a result for a 'static' urban cycle test with optimisation switched on the vehicle passes with flying colours but it is possible that under different dynamic conditions with optimisation switched on it fails miserably and yet, as suggested before, a more realistic average brings it back within the required limits.

    Then you assume/claim that in as much as the system is goal seeking an optimum for any instantaneous set of conditions it is also adjusting its operating or starting point over time to compensate for things like engine wear or component performance changes such as carbon build up, decline in catalytic converter performance or lubrication performance to maintain optimum performance.

    Think along the lines of a neural network whereby it might goal seek a local minima but the true minima is somewhere next door and it needs an occasional kick to find itself in the right place. Again the goal is to achieve an average performance over time that meets the requirements. If you hit it with the wrong specific conditions it fails miserably but overall it passes.

    So... you have a system that over time 'learns and adapts' based on instantaneous adjustments and those that occur over time in order to pass the emissions requirements whilst still giving good overall performance in respect of other desirable factors. For example fuel economy and power output and also operates over the lifetime of the vehicle to maintain such performance.

    You also do not have to be overly concerned about gee whizz maths to attempt to characterise or determine what the answer should be. You just run the car through a 'bucket load of testing' and let the system get on with doing its job and when it is 'there', or near enough, that becomes your production 'ROM'. Bung it in the rest of the cars and let them get on with tuning themselves.

    Of course if your 'bucket load of testing' is largely restricted either by choice or oversight to what the regulatory authorities require, seems highly likely, then your system 'learns' and becomes optimised to those particular circumstances and... all of a sudden, it looks like it is cheating. Also by its very nature you 'do not know' how it does what it does. It just does it.

    Having come up with this super whizz idea/system of course the first thing you do not want to do is 'patent' it. Either every other car manufacturer is doing the same or a similar thing or they are not and you certainly would not want to alert them to your methods...

    I suppose in the first case you would not be able to get a patent, outside of the US, because it is pretty bloody obvious. After all. I am not 'skilled in the art' but I have managed to come up with the suggestion. Also it is software and effectively based on 'black magic'.

    "System that Guesses the Right Answer"??

    I might also guess that if you were to give any hint to the authorities as to what your 'game' was that they would suffer a brain fart short circuit and blibble about demanding that you provide 'proof', 'Turin Complete'?, that this 'splodge' of code actually does what you say it does on its tin but you cannot because you don't know yourselves. It just 'passes the tests'.

    As a result it is much easier to let the authorities run the tests or run them yourselves according to the laid out requirements and print out the results demonstrating that it 'works'. Then one day someone wangs it out of the ballpark and discovers that it fails or finds a set of conditions where it fails miserably and starts blowing whistles.

    Do you fess up as to what you are really doing and give the 'real' game away to everyone else who is either doing the same or are so far behind the curve because they are not or keep quiet and take a hit to your bottom line whilst pretending you have rogue engineers but you are going to sort things out which may or may not, but is likely to, include providing a bigger urea bottle.

    One way you give away your, non patentable, 'secrets' and everyone else gets to benefit from your previously 'secret' technology.

    The other way people are happy that they have a bigger bottle of piss in their cars and the supposed problem has supposedly gone away. In the meantime your "System that Guesses the Right Answer" is still in place, discovers that it has a bigger bottle of piss to drink from and learns how to use it.

    </Flight of Fantasy Mode>

    Of course all of the above falls over if, on analysing the source code, the authorities discover.

    While BeingTested(Params.Rec) = True then else GetYourMotorRunning(nil);

  29. Anonymous Coward
    Anonymous Coward

    All Execs deliberately make themselves ignorant of the facts

    At the top, this allows them to cream off millions in salary, whilst blaming all the failings on the poor bastards below.

  30. Anonymous Coward
    Childcatcher

    Don't bend over to pick up the soap!

    I hope he gets to share a cell with Bubba, up at the 'big house'.

  31. wiggers

    They get around a bit

    These 'rogue engineers' seem to have been working for other companies also!

    http://www.theguardian.com/environment/2015/oct/09/mercedes-honda-mazda-mitsubishi-diesel-emissions-row?CMP=twt_gu

    And in the US in the 90s it seems they were all at it...

    http://arstechnica.co.uk/cars/2015/10/volkswagens-emissions-cheating-scandal-has-a-long-complicated-history/

    1. JeffyPoooh
      Pint

      Re: They get around a bit

      ^- Wrong interpretation.

      The laboratory tests do not reflect the real world. In other words, YMMV.

      Presumably, the test limits were set to what is possible in the lab (likely with urea injection, without cheating). Test limits for on the road testing would have to be set higher, until technology catches up.

      Or go without diesel car sales for a year or two.

  32. heyrick Silver badge

    Either your entire organization is incompetent when it comes to intellectual property,

    Germany? Software parents? Or is this the America Distortion Field in operation?

    PS: Can we have a direct URL for the ElReg server? You're using bloody CloudFlare now so I get so many Server Error, Bad Gateway, and all the rest that only ever seems to affect CloudFlare enabled sites. If your server is wobbly, I'm afraid I'd rather take my chances with the server itself, not some third party.

    1. JeffyPoooh
      Pint

      Either your entire organization is incompetent when it comes to intellectual property,

      heyrick complained about Cloudflare...

      It's you.

      I access ElReg on a half dozen devices and several OS. No errors.

      Post more details, perhaps we can help fix your PC for you.

      1. Camilla Smythe

        Re: Either your entire organization is incompetent when it comes to intellectual property,

        @Jeffums..

        It's you.

        I access ElReg on a half dozen devices and several OS. No errors.

        Post more details, perhaps we can help fix your PC for you.

        I have recently had similar 'problems'. Can you post a copy of your most recent version of commondlg.dll

        Thanks.

        1. Mike Flex

          Re: Either your entire organization is incompetent when it comes to intellectual property,

          "Post more details, perhaps we can help fix your PC for you."

          Saw this Cloudflare problem on my Android tablet this (Saturday) morning. Worked again an hour later.

      2. heyrick Silver badge

        Re: Either your entire organization is incompetent when it comes to intellectual property,

        Post more details, perhaps we can help fix your PC for you.

        iPad Mini with ADSL box, Orange France.

        Android phone (and iPad via tethering), 3G+ mobile network.

        I have a screenshot, that's the best I can do. http://i.imgur.com/gboyFxL.jpg I've also had Bad Gateway a couple of times today.

        1. Mark 85

          Re: Either your entire organization is incompetent when it comes to intellectual property,

          I don't think the ones who can fix this read the comments....

          Send an email to: webmaster@theregister.co.uk with the problem and any screenshots. I've done this when I've had issues and they have responded pretty quickly.

  33. Tubz Silver badge
    Stop

    Cobblers

    I never knew or I was following orders defence didn't work in 1945 or now !

  34. xyz123 Silver badge

    The defeat device was a mixture of hardware, software and the design and shape of the vehicles.

    Because VW is unwilling to admit liability and is instead looking to see if any of their software engineers will take several million dollars in bribes to be the scapegoat, the FULL force of the law should be brought down hard on VW.

    The defeat device cost 10s of millions to develop, it involved re-shaping parts of the car to fit the new sensors, training staff to fit the hardware sensors, software beta testing, etc etc.

    Physically not possible for two people to design and fit 11 million of these devices without having first got expenses approval for the 10s of millions of dollars to develop the device, 10s of millions to pay chinese companies to BUILD the extra sensors, then he's claiming that the two software engineers (obviously at the speed of superman!) were able to fly in and insert the devices DURING each cars manufacture? THEN fly again at ultra-speeds to the emissions testing areas across the world and 'somehow' unnoticed ensure the devices were working as intended?

    This CEO thinks congress is a joke and is laughing behind their backs thinking they're stupid enough to believe anything if it involves 'software' and 'them computer thingies'.

    1. Wommit
      Black Helicopters

      "This CEO thinks congress is a joke and is laughing behind their backs thinking they're stupid enough to believe anything if it involves 'software' and 'them computer thingies'."

      Umm... Individually I suspect that some of your congress critters have a high IQ and deep understanding of current issues and problems.

      However recent US Government comments / desires / actions have shown the world that collectively the US government has a room temperature IQ, and that room is in the Arctic, in an abandoned igloo.

  35. david hill 1

    Volkswagen is just a mere example of how corporate power corrupts and undermines the people at the expense of power, wealth and sheer greed of the corporation and capitalist industrialists/the wealthiest top 1%.

    Médecins Sans Frontières, the humanitarian group of international doctors, nurses and clinicians, has stated that the TTP (the Asian equivalent of the TTIP), would dismantle public health safeguards enshrined in international law and restrict access to affordable generic medicines for millions of people in developing countries. Therefore US politicians and corporate power are driving profits as usual at the expense of humanity and where apparently they just do not care about the human side of things, for wealth, power and greed is all that these people ultimately comprehend. That is why the world is in the terrible dire state that it is, as the mentality is that of an insane human and nothing else.

    The Trans-Pacific Partnership: A Threat to Affordable Medicines for Millions (Médecins Sans Frontières, the humanitarian group of international doctors, nurses and clinicians)

    http://www.msfaccess.org/spotlight-on/trans-pacific-partnership-agreement

    Humanity is Entering the Final Stage of 'its' Development Existence, as Corporate Power and Greed, Consolidated Globalisation and the Politico-Wealthy Corporate Agenda Engages with the World's Largest Ever by Far Global Trade Agreements that will undermine the World order, not Sustain it - http://worldinnovationfoundation.blogspot.co.uk/2015/09/humanity-is-entering-final-stage-of-its.html

    But also this week, the hospital run by Médecins Sans Frontières in the Afghan city of Kunduz was bombed, killing nine MSF staff members together with twenty two patients, and what can be seen as an indicator that the US have other ideals. The reason is that they bombed the hospital for about 1-hour even though the humanitarian Médecins Sans Frontières organization that cares and medically helps all humans, contacted the US to ask them to stop straight after the first hit. They did not and continued.

    Indeed, there seems to be emerging, a sense that the US, through its vested corporate-interest and vast weapons industry (largest producer and creator of killing arms in the world by far), has created wars, not stopped them. In this respect questions are being asked, is ISIS a joint American generated situation to sell US armaments.

    Q. I wonder if Snowden or Assange knows?

    A. Probably, but no-one will believe them where western controlled media is concerned by the censorship of the richest 1%..

    Is the 'Islamic State' a CIA-Mossad Creation? - http://worldinnovationfoundation.blogspot.co.uk/2014/09/is-islamic-state-cia-mossad-creation.html

    Time will tell, but it is going to take millions of lives, human misery and unprecedented intercontinental displacements (more than even WW1 & WW2 combined) to find out. There is a simple question to ask ourselves in the West, "Why do you think that many in the world hate us"? It certainly is not for lack of trying, so what could it be?

    1. druck Silver badge
      Facepalm

      Nurse, more pills please.

  36. Anonymous Coward
    Anonymous Coward

    This is why I urge any engineer/scientist with a brain to shun large multinationals

    Your hard work and ingenuity will go towards enriching scumbag executives who earn salaries orders of magnitude larger than yourself and the moment something goes wrong, you will be directly in the firing, even if you were following explicit directions from above.

    The American executive management model that's all pervasive nowadays really deserves to die an ugly death. Nobody in a company should be earning more than 100 times that of the lowest paid employee and those at the top should bear a proportional level of punishment when a scandal breaks. What was VW Winterkorn's punishment? A €1 million pension plus a €3.2 million golden goodbye. I suppose it's all relative though, because his salary + bonuses last year amounted to €16 million!

  37. The Islander

    amanfromarse Real Culprits

    I'm with amanfromarse on this.

    Mr. Horn's reference to the "couple of engineers" surely both displays his wish to offload a problem and his true respect for those whose contribution is perceived to be purely technical.

    Commentards have jumped to the understandable defence of their own, but as amanfrommarse indicates, this "couple" of engineers likely managed a significant number of engineers between them. And therefore helps to explain how the cheats got into production as people have noted the challenges to be overcome with that.

    This can not detract from the wrongdoing of course, and seems to point to at least a rotten cultural / organisational system in parts of VW

  38. Frumious Bandersnatch

    modern configuration management systems are wonderful

    They let you trace back exactly who checked in a given code set/patch.

    Management often wants to have scripts running on top of this to link changes to requirements docs or other change requests. I imagine that there's a very embarrassing virtual paper trail stored somewhere. Unless something bad "accidentally" happened to the servers and all the backups ...

    /git blame

    1. Will 28

      Re: modern configuration management systems are wonderful

      You can sneak code in if you really want. I don't believe it happened in this case, but a driven developer is usually able to smuggle a change through without being spotted. It can be as simple as using a build server account, or even as unfair as just accessing a colleagues machine (often via a share that the naieve colleague has agreed to) and altering some files to piggy back a change on a legitimate changeset.

      I don't think it happened here, but I don't agree with those saying that a good change control system could prevent this 100%.

  39. Stevie

    Bah!

    Well, this "two software guys who we haven't found yet" theory sounds convincing to me.

    I'm sold.

  40. Grumpy Fellow
    Meh

    Maybe, maybe not

    As much as I think the chief is lying, I have to say that I've been doing embedded software since the 1980s, and for most of that time I was the only one on God's green Earth who knew what most of the lines of code that I wrote actually did. Sure, we had peer review and SQA and audits, but If I wrote a line of code that turned off the Clean Mode and labeled it with a // Enable Clean Mode comment, then it would make it past the reviewers. Especially if there were pointers involved and maybe an enumeration in the mix. It is hard enough to detect unintentional errors in embedded C code. Finding an obfuscated intentional "error" is pretty unlikely. So, if I were doing embedded code for VW, why would I turn off the emissions controls? Maybe I think that the emissions rules are BS and that our customers deserve a better running car (not my personal opinion, mind you). Maybe I do it so that I earn the respect of my fellow engineers, or so I get to keep my job in a downturn because I am the Diesel Engine Wizard at VW. Embedded software doesn't have to be a black art, but in most organizations it is still practiced within a narrow slice of the organization without much visibility upward.

  41. Adam Azarchs

    Design reviews

    If you've ever worked at a company anything like a car company, you know how many reviews and design needs to go through before it's put into production. A lot more than "a few" employees would need to have been in on it, and if no one from such a large set felt the need to run it by a lawyer I'd be shocked. Far more shocked than if I heard the lawyer was told, offered an opinion, and was overruled.

    1. Wommit
      Pint

      Re: Design reviews

      "... the lawyer was told, offered an opinion, and was overruled." And then the bean counters stared into their golden abacuses and said "If we fix the problem now it will cost us €xM, and everyone will laugh at us and call us incompetant. If we leave it in, and nobody notices we've saved €xM. If we leave it in and then someone notices, then we blame it on 'rogue engineers', and it will cost us < €xM."

      There, finished it for you.

  42. Florida1920
    Headmaster

    Management Strategy 101

    First, fix the blame. Then fix the problem.

  43. 0laf

    [Sarcasm]Wow, totally didn't see this coming[/sarcasm]

  44. 404

    He knew.

    Engineer coder says, 'Hey check it out, if we do this, this happens'. Manager says 'Neat', sends it up the line. Management/Marketing says 'Hrrr Hrrr'. Implemented.

    All of them probably giggled their asses off thinking how smart they were.

    Bastiches...

  45. Anonymous Coward
    Anonymous Coward

    Corporations pay execs huge bonus's when the company does well but when things go wrong the company appears to be run entirely by ghosts and computers so they pay a fine for crimes.

  46. Cincinnataroo

    Good example of sand in the face

    Sure VW did bad. Stupendous, organised, idiotic, bad for profit:

    BUT

    They were being watched (and people paid for those emissions tests) by employees of nation states. These employees of the states and the people they represent didn't notice this for years. Even when the scientific results came in (April/May 2014) nothing noticeable happened immediately. Don't let sand be kicked in your face. Much more is rotten and useless here than VW.

    If they try to put a noose around the neck of a scapegoat or two, those scapegoats (and those who know the truth) should spill absolutely all the beans and not lose any pension... for doing that. Come to think of it details should be spilled irrespective of scapegoating.

  47. simmondp

    Engine software (firmware) development is tightly controlled - to add a "feature" will have been tightly controlled with a specification, writing, testing and QA. All stages will have required approval and sign-off.

    Thus is beggars belief it was "just a couple of rogue engineers".

    If it really was a rogue addition then you should really worry about the whole production process and the rigour with which VW cars are produced.

  48. Kleykenb

    rogue CEO

    "Rogue employees" wouldn't do this kind of thing without telling their superiors. You'd want to get promoted for that kind of shit or get a raise.

    No. he's dishonest and hopefully he'll answer for that later on.

  49. DerekCurrie
    WTF?

    So what about the rest of the diesel car industry that ALSO pull this scam?

    Did those two renegade software developers at VW write the code for ALL THE OTHER diesel car companies that equally abused their customers? I don't think so. I think this situation doesn't just go to the very top of VW. I think this trick was shared among the top executives of ALL of these companies and became a matter of competition. If your scum company didn't scam the customer, they were going to lose out.

    Result: A corruption eddy. You're ALL to blame.

  50. LucreLout
    Mushroom

    Really??

    Look, guys, you lurk at the top of these organisations hoovering up vast compensation profiles for doing what is in reality very little work. You lay personal claim to all the glory as justification for your fantasy land remuneration.

    And then it goes wrong. The least you can do is hold up your hands, show a bit of self respect and dignity, and have the least little bit of decency to the companies workers, by taking ownership of your failure.

    There will be fines. There may be jail. Play for bigger prizes and the penalties also increase. T'was ever thus. You gambled and lost, now settle your debts like men.

  51. kmac499

    Easter Eggs and Stinking Eggs

    Easter Eggs of one form or another have been in software for years along with hidden modes for development testing, or just plain old convenience of the people building systems.

    Once in it's easy to isolate them, but not always easy to remove them. It's even conceivable that such test modes were precisely for simulating the testing cycles of engines under development.

    I can totally believe that 'a couple' of software engineers could inroduce such a system and I can also belive that it may have been originally added as quick fix because "We know we can improve the engines perfomance later so JFDI it for now." and we all know that second opportunities never come around.

    What I also know is that system\code reviews and QA will only review what's shown them. Whether this be approving the algorithms for a design, or testing new expected features and the reversion testing to make sure nothing existing is broken. Rarely, if ever, is code looked at line by line.

    What some people here seem to be expecting is some form of forensic testing on the assumption that fraud may be present. That level of supsicion may be routine in financial sytems but as I assume the primary concern of Auto Engineers is safety, their testing would ensure the driver retained cotrol of the vehicle at all times. (Memories of the FADEC software on Chinook helicopters)

    If this proves not to be a deleberate and authorised attempt to deceive, but a misguided attempt to hit targets then maybe senior managers should change the cultures they encourage.

    1. Anonymous Coward
      Anonymous Coward

      Re: Easter Eggs and Stinking Eggs

      "Rarely, if ever, is code looked at line by line. What some people here seem to be expecting is some form of forensic testing on the assumption that fraud may be present. That level of supsicion may be routine in financial sytems but as I assume the primary concern of Auto Engineers is safety, their testing would ensure the driver retained cotrol of the vehicle at all times. "

      If you'd read any serious coverage of the Toyota "uncommanded acceleration" case, you'd know that system safety seemed to come somewhere down the list of priorities. See e.g.

      http://www.eetimes.com/document.asp?doc_id=1319903

      Meanwhile, back to VW. You and others might want to go read about MCDC testing (it's in Wikipedia and elsewhere, it's not rocket science).

      It's compulsory on FADEC systems used in airline engine controls (per DO178) and highly recommended on car engine controls (per ISO26262).

      The implications include the following (grossly oversimplified):

      * you know what every code block is for and can show that every code block has been executed

      * you know where every decision point is and can show that every option in every decision point has been exercised

      * you have a set of documented test data that repeatably shows the above conditions are met

      There's more, but that'll do for here.

      Additionally there should be an audit/review process that ensures the necessary process requirements have all been met.

      So for the VW situation to arise, we probably haven't met the above conditions, and the audit/review process hasn't noticed that the conditions haven't been met?

      And I write this as someone who historically has had limited faith in DO178 etc.

      Either utterly implausible or utterly incompetent. Not good for VW either way.

  52. John Sawyer

    So much for German exceptionalism

  53. Halfmad

    Doesn't bode well..

    With the increasing reliance on wireless and "connectivity" between devices if they aren't doing internal testing that would highlight this sort of flaw surely they should be reconsidering how cars operate and connectivity until they have some assurance about their testing regimes?

  54. cs94njw

    As a software engineer, I would be excited to add extra functionality to a dashboard, or tune the engine efficiency.

    If I made a change which dramatically reduced the emissions, the engineers are gonna want to know what I'm doing on their "turf", and what the hell I did.

    But where's the incentive for it?

    The software engineers (instead of the hardware engineers) really have a KRA to reduce emissions by 300%?

  55. Geoffrey W

    Next step in the investigation is to find a culprit who is conveniently dead

  56. shaunhw

    I'm not trying to deflect blame from the vehicle manufacturers, but I can't believe these "regulators" didn't ensure the emissions were properly tested under real driving situations. The vehicles obviously passed their completely inadequate tests with flying colours to begin with. Was that all that was actually required ? It was all the regulators required to accept the vehicles were up to standard.

    So don't the regulators and authorities deserve some small amount of blame here for failing to make their tests a little more foolproof ? I am sure this wouldn't have been rocket science by any means, and they clearly also failed their part in this fiasco.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like