back to article Top VW exec blames car pollution cheatware scandal on 'a couple of software engineers'

Volkswagen America CEO Michael Horn has played the "rogue employee" card to explain how and why his cars' engine software cheated in pollution tests. While being grilled by the US House Committee on Energy and Commerce on Thursday, Horn said he only learned of the existence of the so-called emissions-hoodwinking "defeat device …

I can readily believe that it was the actions of a few rogue employees. I've seen it happen.

But if VW hasn't identified who those employees are this far along, they are incompetent or lying. This is a major incident that may cost billions of dollars. VW should have people working 16 hour days to figure out how this happened and how to mitigate the damage.

3
43

There's also the fact that the VW Group CEO has already stepped down - you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up. This is Horn trying to save himself.

53
7
Silver badge

I agree...

...there is a worrying lack of appreciation of how intense reality distortion fields can be and how much rationalisation can be manufactured to get past an: "I can't tell the Boss\Customers\Shareholders THAT! He \they won't like it!! <cringe>" culture.

I can buy the idea that VW execs. didn't know about this because I can buy the idea that they created a culture where "positivity" was encouraged (promoted) and "negativity" was discouraged (fired) to the point that they were left bouncing around in an echo chamber of their own wishful thinking. I've seen it happen first hand. That sort of ignorance is still culpable of course.

44
0
bri

@Esskay

Au contraire, you do step down to appease shareholders and other relevant parties, when such a massive loss of value occurs. And I don't believe that Winterkorn wanted to clean up the mess created under his watch, with everyone questioning his every move.

Horn is in full corporate damage control mode, as minimizing fallout is crucial to future survival of the company. I don't believe a word he said, but I fully understand why he said it. He also has no other choice. Unless he wants to endanger hundreds of thousands employees. The are no good choices for VW now.

10
4

" you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up."

Wasn't he due for retirement? If I was a CEO and had my retirement lined up, I'd retire to go fishing rather than face a shitstorm.

30
0

Other makers

I wonder what the story at other manufacturers will be, since I doubt VW will be the only ones making use of technology to make their cars appear better to the buying public.

If your competitor appears to be making cars that are so much better than yours, in terms of fuel efficiency at least, then what do you do?

Get consigned to history or see what you can do to catch up??

7
0
Silver badge
Mushroom

Re: Other makers

Or possibly be honest built it to spec and take the hit on your bottom line. There seems to have been a strong whiff of the profit motive about VW's actions - I suspect it was initially about building to a price for the American market - then it just grew by habit. I suspect they were trying to undercut BMW and Mercedes in the US market- whilst trying to also promote diesel as the fuel of choice as a differentiator. Up to now the VW TDI's have had a certain level of caché that in trying to capitalise on they have flushed down the toilet.

I hope this "2 engineers" comment encourages one or more of them to turn states evidence and shaft the squirming CEO's good and proper.

I find it intensely annoying that the default mode of any corporate drone these days is to lie, prevaricate and deflect instead of just telling the truth. There need to be much stronger laws on corporate perjury. i.e. both jail time for the Exec and % of world turnover fine for the company.

17
0

Re: Other makers

Indeed, Lance Armstrong knew that feeling.

2
0
Silver badge

Re: @Esskay

@bri

"This was not a corporate decision from my point of view," Horn said while under oath.

"I don't believe a word he said"

Neither do I, but Horn believes it. Just like George Costanza said, "It's not a lie, if you believe it."

Rogue employees? My ass, that's almost as bad as "some older boys made me do it." "A misinterpretation of a directive by "a" team in middle management are responsible." would be a much more plausible way to focus and defuse the guilt elsewhere.

It will be interesting to see the testimony given in the US vs the testimony given in Germany. I get the feeling they won't be the same story.

10
0

@Esskay

> you don't step down as group CEO because a couple of software engineers in the US subsidiary of your company fucked up

And anyway, you have a nice golden lifeboat with a lucrative retirement plan set up...

2
0
Silver badge

Re: Other makers

Every automobile manufacturer that I dealt with in the 80-90's would purchase a few of each of its competitors vehicle as soon as they hit the road and investigate many aspects of its performance. I find it very hard to believe that the VW trickery wasn't known within professional engineering circles in the engineering end of the business.

Getting clean diesel engines to work is hard - it's unbelievable that other manufacturers didn't say, "Let's take a look at what VW are doing"

17
0

Re: @Esskay

>He also has no other choice.

Let's see what happens when a few staff are fingered and they decide to leak emails from management.

Of course it could have been a few rogue coders who just happened to write code that didn't need to be tested physically, and didn't produce any obvious unwanted side effects.

Of course it could.

Classy move, anyway. I hope he goes to jail.

10
0

Rogue employees...

This smacks of a business decision made by marketing & sales; not a decision made by engineers.

I have no doubt that the engineers came up with the implementation of the solution, in exactly the same way that I'm certain that the engineers were not the ones driving the bus that they would eventually be thrown under.

13
0

Re: Other makers

"Getting clean diesel engines to work is hard - it's unbelievable that other manufacturers didn't say, "Let's take a look at what VW are doing""

Indeed, modern diesel engines are much more expensive to develop than their petrol driven counterparts, so car manufacturers often cooperate in their design and development (I know of at least one British manufacturer that developed several diesel engines with a French competitor.)

It does therefore seem a little unlikely that only VW know how to do this slight of hand with their engine emmissions. Whether others have put this know-how into practice as well, I guess only time (and screaming headlines) will tell.

2
0
Silver badge

Re: @No good choices for VW

The "all the fault of a very few engineers" argument was aired at the same time as the CEO left. If that is really the truth I will never buy another VW because you cannot trust the factory to make what it thinks it is making.

VW has had enough time to find a solid chain of command from programmer to board level (all of whom, rather surprisingly, would be getting on in years and very fond of fishing). That scenario might not be the truth but it doesn't stink to high heaven like the current one. There are far better choices than choosing an indefensible defence.

2
0

The engineers involved would be in Germany, since the design was mainly done there working with Bosch, who supplied the pollution control system components. TDI diesels with the pollution control hack were shipped to a lot of countries, including Germany itself. It might not have required that much work on the part of VW engineers since it looks like it took advantage of (presumably) test modes in Bosch' own equipment - simply turned various features on (or off) to get the desired reduction in pollution during emissions tests.

An article in the Telegraph states that Bosch sent a memo to VW in 2007 warning that use of the SW to defeat emissions test measurements would be illegal. Wonder if some VW engineer made an incautious comment in front of a Bosch representative leading to that warning.

9
0

Re: Other makers

I saw a comment in one of the early articles after this broke that there were "a lot of questions" around how VW could get the claimed combination of power, fuel economy, AND low NOx pollution from a relatively "primitive" pollution control system. I suspect no one in the industry wanted to blow the whistle lest questions be raised about THEIR results.

I've already heard another question about a somewhat less elaborate hack in a similar vein - control of urea injection designed to insure good results in test, but stop using it in regular driving to avoid drivers having to refill the urea solution too often (or at all).

1
0
Silver badge

Or, Bosch knew that VW was going to activate the test mode memo was merely a CYA memo.. The thing is, why wasn't the test mode deleted from the delivered code if it was "illegal"..?

0
0
Silver badge

Re: @bri

I agree, nobody would have confidence in Winterkorn, whether he had known or not. With a "clean" start, with a new CEO, there is less residue for people to point fingers at. (Sorry for the puns).

What got me was this: Rep. Chris Collins (R-NY), himself a former engineer, noted that if this defeat device was the work of two Volkswagen engineers acting alone, the company's intellectual property department would have every reason to seek to patent the innovation unless they were directed otherwise by management.

And if he knew anything about Europe, he would know that you couldn't patent that anyway. That is software, software isn't patentable.

4
1
Silver badge
Boffin

@ Esskay

What rock have you been living under? The software cheat originated in Germany, which is why it was applied WORLD WIDE. VW got caught in the US because US environmental requirements for autos are tougher than anywhere else as is testing.

1
3

Re: Other makers

"I find it very hard to believe that the VW trickery wasn't known within professional engineering circles in the engineering end of the business."

That's when the conspiracy theory kicks in...

0
0

Re: I agree...

Robocop 2 had it covered perfectly:

Minion: It'll be a feeding frenzy.

Boss: It's only money. What about criminal proceedings?

Minion: Well, we're looking at major indictments, sir. Prison terms.

Boss: What about me?

Minion: You know we'll do whatever we can.

Boss: That's not good enough.

Minion2: Sir? What if this was the work of one individual? A person who had her own agenda, wasn't in sync with the goals of our company?

Boss: Well, that usually works.

Minion2: A woman who was not a team player, who violated our trust.

Boss: Well we'd need some evidence to support that.

Minion: Sir, whether it exists or not, I know I can find it.

5
0
Silver badge

I can readily believe that it was the actions of a few rogue employees. I've seen it happen.

Really?! A significant innovation comes through, and no managers whatsoever know either who came up with the idea, nor sought details in order to patent it?!

Yes, rogue engineers of all kinds sometimes slip something under the radar. But this wasn't slipping a back door in or a fudge to work around or hide an issue they couldn't fix, this was a blatant cheat which produced a clear competitive advantage over their rivals. If it had happened and noone in management had known, the engineers (or their dept, or their manager, or his manager etc.) would have had praise heaped on them for saving the company so much money and/or giving them such an edge over their competitors. They would also have immediately asked for the details so that they could patent it before their competitors could copy it.

There is simply no way that knowledge of this could not extend at least a fair way up the management hierarchy. At the very least, there would have been a point where a manager asked about it and was told "it's better that you don't know...".

2
0
Silver badge

Re: @Esskay

"This was not a corporate decision from my point of view,"

Since the very moment this shitstorm broke I was expecting the moment that some 'rogue employees' were blamed. Predictable as night follows day.

1
0
Silver badge

"The thing is, why wasn't the test mode deleted from the delivered code if it was "illegal"..?"

Because it's needed to stop the car panicing when it's on a rolling road.

0
0

Re: Other makers

"...purchase a few of each of its competitors vehicle as soon as they hit the road and investigate many aspects of its performance.

Well, everyone does that, but the primary focus is what 'the others' do to keep the production costs down.

However, if they think that Manufacturer A is doing a better job with, say, Diesel engines than they are, Manufacturer B will give increased attention to anything that manufacturer A is doing.

That said, this is way more difficult than, say, 'Why are their seats cheaper than ours?'.

But, yes, I would say that it is difficult to believe that at least some other manufacturers didn't have a very good idea that something odd was going on, even if they couldn't describe what the strategy was, exactly. And, how did the University researchers know to look at VW? Good luck, or did someone give them a word to the wise?

0
0

Re: I agree...

"That sort of ignorance is still culpable of course.

Exactly, although 'our engineers will do the impossible by next Wednesday' is worse.

0
0
Anonymous Coward

In this instance, there's no way it was due solely to the actions of a few lowly engineers. It could only have worked at the scale it did with executive buy in.

There is substantial documentation showing VW (at the highest levels) have been officially notified repeatedly for years about problems with their emissions. They've also previously been fined for rigging results, so this isn't just a one-off.

It's unfortunate their execs feel lying is the correct path to take here. It's not unexpected though, as modern Western style management (US MBA driven) is all about short term thinking, corporate greed, and PR spin. :(

52
1
Silver badge

So, even when it's Germans doing the lying, make sure everyone knows that the US is still the guilty party, eh? You sound like a Democrat blaming 7 years of Obama failure on the previous president.

5
49
Silver badge
WTF?

"From my point of view" ... WUH?

Would these engineers 1) develop, 2) install 3) test, 4) QA test 5) run emission tests 6) write off roll-out?

Maybe VW software development consists of three dudes behaving like a web development startup with a subversion repository and an "it creates HTML, ship it!" culture, but I don't think so.

The idea of patenting an "defeat device" sounds daft though. First, is there anything to patent. Second, patenting it would mean a traceable item discoverable by anyone. "Procedure for quickly killing your wife using a bathtub". Nah, I don't think so.

22
1
Bronze badge

You sound like a Repubican blaming 7 years of Congress failure on the President.

29
3
Anonymous Coward

Re: "From my point of view" ... WUH?

"The idea of patenting an "defeat device" sounds daft though."

He perhaps did not express it very well but he is right. If management was told that a software change had fixed an intractable engineering problem, saving VW vast amounts of money, they would have wanted to patent the method.

If Horn thinks he is telling the truth, then the problem we are seeing is that senior management doesn't understand how engineering works in their own company. That would be a first for Germany.

21
1
Anonymous Coward

Re: "From my point of view" ... WUH?

perhaps, but they'll go that way if VW gets away with this. Plausible deniability. Hire a bunch of MBAs to run things who don't have an Engineering degree between them and they can claim anything at all.

3
0
Silver badge
Unhappy

Re: "From my point of view" ... WUH?

> That would be a first for Germany.

Sadly, not a first. Dunno about VW, but I have witnessed incredible technical ignorance at CXO level in tech-companies.

6
0
Syd

Re: "From my point of view" ... WUH?

Pretty sure he wasn't suggesting they would patent the defeat device - they would patent whatever it was in the engine which made it run much cleaner than any of their competitors for the same performance.

11
0

@david 12

> You sound like a Repubican blaming 7 years of Congress failure on the President.

... when you've done everything in your power to block, damage or subvert everything that that President has tried to achieve.

F(inshed)TFY

10
3
Silver badge

Re: "From my point of view" ... WUH?

I think he means VW group was the only manufacturer to get low diesel emissions without AdBlue. That should have been patented but it wasn't, something that with hindsight stinks from a mile away and not just because of the contamination.

7
0

what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?

4
2

You don't sound like a dumb septic, you are a dumb septic. Chip on shoulder much?

0
0
Silver badge

> "You sound like a Repubican blaming 7 years of Congress failure on the President."

Does that include his first three years when Obama's party dominated Congress? Otr the six years when they controlled the Senate? Pathetic.

4
8
Silver badge

> "what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?"

Both of them "fake but accurate," right?

The unemployed class has expanded dramatically under Obama, but those are no longer counted as unemployed because they've given up looking for work. Still, those like you may still revel in a "low" unemployment figure, at least.

About the deficit, you ought to google "us deficit timeline" if you want a bit of a shock. And let's not even talk about the obscene debt expansion Obama's party has laid onto future taxpayer's backs (when they controlled Congress and spending completely).

But hey, all that massive new spending was at least used for vote buying thru Socialism and Eco-cronyism of the "shovel-ready" kind, not wasted on anything frivolous like defense. Who needs a military when Obama is such a great international statesman?

4
8
Silver badge

Re: @ Graham Marsden

> "... when you've done everything in your power to block, damage or subvert everything that that President has tried to achieve."

Oh yeah, I forgot that the GOP was able to stop Obamacare. Not a single one of them voted for it! Oh wait, that didn't stop Obamacare.

Well at least they stopped him from letting our military be hollowed out by neglect and PC inanity. Oh wait, they didn't stop that either.

Well, there's all that new spending he wanted to pour into green energy...oh yeah, he got that as well. Too bad it all just diappeared down the rat hole.

I could go on and on. Basically the GOP leadership has been Obama's BFF, which is why we now see a grass-roots revolt against those leaders.

There is one thing the GOP did accomplish against Obama; The electoral results of the Obama Years has been utter devastation for the Democrats at all levels except the Presidency, and that won't last much longer either.

3
7
Silver badge
Pint

> what part of the obama admin don't you like, the drop in unemployment, or the drop in the deficit?

AWOOGA! AWOOGA! We have an Alternate Reality Intrusion! WHERE IS DOCTOR WHO?

1
3

And you sound like a Republican blaming previous administrations' failures on the present administration. Tit for tat.

1
0
Anonymous Coward

Even if it was down to "a couple of software engineers"...

This would have to be attributable to unnecessary requirements, as if it was in the requirements it would be a project decision sanctioned by management.

If there are unnecessary requirements, then the project does not comply with ISO 26262, which would also make the problem a management failure.

The argument that it's down to rogue developers can only be supported if the software has been "hacked" without a proper development process being in place.

23
1
Anonymous Coward

Re: Even if it was down to "a couple of software engineers"...

[No apologies for length. Share and enjoy.]

It would be very surprising if the VAG incident comes down to "rogue engineers".

Analysis of similar "engineering vs commercial pressure" precedents typically shows that occurences like this come down to "management culture" (not even "rogue management", just the routine way an organisation does its business).

Even if an individual engineer does something stupid, the *organisation* is supposed to have a set of processes (and in some cases even "corporate ethics") to stop the stupidness reaching the outside world. Realistic system testing, proper QA processes, and such like tend to be helpful but have an unfortunate tendency to be seen by HQ as "non value added".

When the company loses tens of billions of dollars because such "NVA" tasks are skipped, maybe it's time for HQ to rethink that viewpoint.

I mention "corporate ethics" because large companies I am familiar with claim to have formal corporate ethics programmes, whistleblower hotlines, and such. I'm afraid that experience in many organisations (private and public sector) suggests that these ethics hotlines etc exist for cosmetic reasons only.

One organisation I knew had such an "ethics programme" but there was also an informal but widely understood and very clear "don't rock the boat, or else" policy: senior engineers/managers do not make mistakes or misjudgements. To even think aloud about the possibility, when a well-informed independent outsider would likely be asking "why is this approach a good idea, show me the documented arguments in its favour", would be a career-limiting move.

Below is a very brief extract from a presentation given to an Oil and Gas Industry conference on some cultural aspects of the 'modern' large-organisation engineering process. The presenter is not an engineer but a man with a history of investigating how disasters happened. Do you think his thoughts might be relevant in this picture?

SEVEN THEMES OF NIMROD

11. The following seven themes struck me forcibly as I began to investigate the Nimrod story:

(1) Complexity. The Byzantine complexity of the organisation, the rules, the standards, the safety processes in the MOD was remarkable. Complexity and change had become the altar at which a lot of senior people worshipped – but had become the problem rather than solution.

(2) Dilution. The immediate casualty of this complexity was a dilution of responsibility and accountability – it was difficult to divine who was responsible or accountable for what. Accountability is the ‘reciprocal’ of Responsibility.

(3) Management by committee and consensus. There were more committees, sub-committees, working parties etc dealing with safety-related matters than the UN.

(4) Lack of challenge. There was a culture which rewarded conformity rather than the asking of awkward questions.

(5) Migration. There was a migration of decision-making and budgetary power away from those with most direct working knowledge to those sitting in warm offices back home.

(6) Triumph of generalists over specialists. There was too little appreciation of engineering specialist skills, too great a reverence to for the young MBA.

(7) Paper safety. Safety was increasingly a paper, coloured diagram and PowerPoints exercise, rather than a people, process and cultural matter.

From Charles Haddon-Cave's presentation on "LEADERSHIP & CULTURE, PRINCIPLES & PROFESSIONALISM, SIMPLICITY & SAFETY" at a 2013 conference marking the 25th anniversary of the Piper Alpha oil rig disaster.

https://www.judiciary.gov.uk/wp-content/uploads/JCO/Documents/Speeches/ch-c-speech-piper25-190613.pdf

2
0
Thumb Down

Just when you think they're in deep enough already they lie their asses off again. The whole company really has a sleazebag culture, apparently.

17
1
Silver badge

There are Lies coming even from their board.

8
1
Silver badge

Software Code Reviews?

It doesn't say much for their software management processes. If you've got a couple of engineers with an idea, they have to spend time developing the algorithm to recognise and defeat the test, then check it out by seeing if it's capable of recognising test conditions reliably enough to be useful. All this takes time and resources, which presumably the management didn't notice?

Then you have the formal code review before you release production code. Generally this involves letting other people look at your source code, may require you to have all your curly brackets in the correct place and use the correct combination of tabs and spaces and possibly even go through the functionality of the code and check out the results of your regression tests. Then you get the reviewers to sign off that it's OK and does what is required. All without them noticing the big block of source, the function of which wasn't clearly explained.

Yes, clearly the work of a couple of rogue software engineers who are capable of amazing feats when it comes to hoodwinking the system.

50
1

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018