Dell is warning customers that there is malware on some of its server motherboards. The PowerEdge R410 Rack server has spyware within its embedded systems management software. The direct seller is sending customers letters warning of the danger and also telephoning those affected. A post in a support forum says customers …
You don't expect your hardware supplier to be embedding spyware in it's products. At least not accidentally. I hope they offer a full explanation of what happened and how, in addition to whatever apologies/refunds are deemed acceptable by anyone who actually bought this stuff.
Maybe there is a clue here:
"He said the malware would not infect non-Windows servers."
So Linux won't have a problem and Windows does. Sounds like a perfect reason to change.
Because of course..
...it would be totally impossible to incorporate Linux malware the same way.
Missing the point
Top prize there for missing the point!
This has sod-all to do with Linux. It actually has sod-all to do with Windows either. It has to do with what the dickens are Dell doing to allow this to happen!!
f'ing linux zealot comment
Not quite my eager friend
Back in the day the firmware used to be simple, there was no room in ROM and no reason for a full blown O/S to be running the on-board management system. Now with bloatware management systems, this is the upshot.
Linux or Windows, makes no difference! The secret my little zealot friend, is in the fact that the O/S used is consistent build, thus any code running on it can easily be run against every system board running this tragic on-board firmware/management software base.
Bring back the ZX Spectrum coders, the miracles they could perform in 48K ( or even 16K ) of RAM would send these modern, so-called coders, back to school to learn how to write code properly!
Too many fanboi's foaming at the mouth on this... The issue is in the 'Management' software. So one would imagine that it happens outside of the OS.
And to your point... how did it get in there in the first place?
Where are the 410s built? (Motherboards.)
Who has access?
Not at all a Linux Zealot comment
How does malware in the firmware affect a running operating system? Presumably because the operating system calls the BIOS code.
Linux famously ignores the BIOS, probes andcontrols all the hardware itself. There's a Linus diatribe out there about the sorts of "programmers" employed to write BIOSes being lower in the food chain than pond slime. Looks as if he was right (again)!
I can't help thinking that things went astray on the day that a general-purpose computer acquired more firmware than that needed to load block zero of a mass-storage device and jump into it. Create a big enough address space, and nasty things will take up residence therein.
Guessing, but it's in the system management BIOS, that runs around the real BIOS and O/S. It's meant for remote administration and so on, but is a security nightmare - as recent posts by the folks at Invisible Things should tell you...
Probably made by the lowest bidder om hardware....
which of course is probably in China. So why not have the state slip in some interesting goodies into the BIOS.
@ Nigel 11
I wonder if this is related to the change to EFI "firmware" from conventional ROM BIOS? My eeePC has an EFI partition. It's something like 32Mb big. WTF? Just to set the clock and boot options using a DOS-like interface?
Do you have a link to the BIOS programmer diatribe? Sounds like an interesting read.
@Herby - Sounds like a reason to change alright - change to Proliant.
@Nigel 11 - Windows "ignores the BIOS" too, yust as soon as the system has been bootstrappede - like linux. you'll notice there aren't any boot sector viruses any more? What is probably happening here is that the systems integration software in the OS is talking to the malware ridden firmware - Therefore, this could happen with Linux too.
A short version: http://lwn.net/Articles/387610/
"We do not trust BIOS tables, because BIOS writers are invariably totally incompetent crack-addicted monkeys. If they weren't, they wouldn't be BIOS writers. QED. And in fact the Apple problem is an _example_ of this BIOS writer incompetence, not some shining example of them doing something right." (Linus)
I'm sure I saw him expressing similar sentiments at greater length somewhere else but I can't find the link.
The same way the ground affects a building.
System firmware operates at such a low level that, frequently, it can do things to the OS that the latter can do nothing about. Beware anything that may, under any circumstance, have unimpeded access to memory. For example, I happen to remember there being a cute little hack for Openboot PROM systems*. Basically, you figure out where your shell's process resides in memory, look at the system headers to learn where the UID is, jump into the firmware console, change the UID to 0, resume execution of the OS, and find yourself sitting in front of a root shell. (Of course, this requires access to the system console and the absence of a PROM password.) Of course, this is just an example of what sort of things the firmware can get away with. I would not expect the aforementioned method to be very closely analogous to what malicious firmware might do, since malicious firmware would not do well to rely on anybody pressing stop+A or similar. No, there's an easier way than that.
You mention that Linux goes out of its way to control the hardware directly, but it has to /boot/ at some point. As anybody knows who's ever had occasion to type "init=/bin/sh" or invoked a conversational boot in VMS, operating systems are (necessarily) quite vulnerable to anything that can interact with the booting process. I would think that this would be the first line of attack for malicious firmware.
Back when I were a kid, all it needed to bring up the system was:
CLD, LDX #&FF, TSX, <poke some hardware into submission>, CLI, JMP <start>
Typically your average OS would bring up the system, initialise video, print some info. Got a harddisc? Something to autoboot? No? Got a floppy? Something to autoboot? No? Got a network? Something to autoboot? No? Fall into BASIC, let the user sort it out...
I mean, seriously, WTF? How does any even half-way professional software team manage to allow malware to embed itself into such critical software? FFS....
My boss just had Kittens
We've just placed 4 Dell servers onto our network ...
He went white as a sheet when I started reading this out loud.
He's calmed down now.
1) they're not R410's ... they're newer.
2) we don't run windows servers ... :)
Do they mean that one of their techs has added the wrong firmware module when programming the chips and some inquisitive sysadmin found it and reported as malware?
There may be trouble ahead...
I used to work for a PC manufacturer and a huge amount of effort went into malware prevention. It was challenging because there were so many ways for it to get in.
The only major outbreak we suffered was caused by a salesman from a well known TV-advertised finance/insurance/pensions company who wanted to see our internal phone list. To get it he bribed one of our receptionists with PC games for her kid. She tried out the software at work and it was virus-infected, so we caught a cold. The low-life salesman was banned from all our sites after that:-)
Was the receptionist fired? She should have been.
No she wasn't fired, but the incident highlighted shortcomings in training. She was hard-working and usually very good at her job. She was absolutely horrified by what had happened and she knew giving him the phone list was a serious mistake. I expect she got a reprimand.
i would think the conversation went like this...
here is X amount of money, put this on the servers... ok, no problem.
Is some one trying to do them in?
First their own capacitor hardware supplier sells them defectives a few years back then this. Spooky.
That's some shiny kit Jad
Newer than an R410? Newer than current?
I wonder if its like the computrace Lojack software that you get on Dell laptops.
I was going bloody spare the other day trying to find out why this "malware" kept reinstalling itself on my system, did my research and identified that it was really an authentic piece of software,
Herby, you're a twat!
Whilst not being a Windows evangelist, it is comments like your that put people off going the way of the penguin.
If you haven't got anything useful to say; keep it closed!
As has been said, it's a Dell problem.
I mean come on, Herby, if your Kung Fu was even half way good, you'd have specified the distro or said something with far more kudos such as i/OS.
Grow up, then we'll allow you to play with a proper computer. Or does your mum not let you out of your bedroom for fear of your hurting yourself?
Sorry for the rant everyone else. I'm just sick and tired of these dickheads banging on about Linux and how good it is. Each OS has its merits and faults dependent on the application for which it is used.
And Herby, yes I have used a fair few of them for the past 25 years from MS-DOS 2.11 on through various flavours of Unix, Windows, OS/400 and MAC/OS - so I think I know what I'm talking about.
You use a several techniques to defend your rant, which I would summarise as sophistry, however in detail
You didn't deal with th inconvenient truth in "not quite my friend" - unhelpfully pointing out that Linux sidesteps the BIOS
"Each OS has its merits and faults dependent on the application for which it is used"
non-popperian - the statement is unfalsifiable thus generally considered to be content-free. In what way, do you think, Linux is an issue in these circumstances? In which ways, do you think, it is advantageous?
"I have used a fair few of them for the past 25 years from MS-DOS 2.11 on through various flavours of Unix, Windows, OS/400 and MAC/OS - so I think I know what I'm talking about."
That might or might not be true but we don't know because you don't tell us, you merely assert that <statement A> gives you authority to make <statement B>
More examples here: http://queinnec.perso.enseeiht.fr/proof.html
I would conclude that your comment contains less information that Herby's but there you go.
Ok, I'll counter...
Firstly, I never stated anything with regard to "sidestepping the BIOS". I think you may find that was another poster...
Each OS has it's merits and faults by virtue that you wouldn't try and play Crysis on a iSeries now would you? Nor would you use Linux for video editing. Would you attempt to run a secure web server using Windows?
Are these merits or a faults. Now, we can get all philosophical about this if you like? However, the plain fact of the matter is that one OS is better suited to the above examples than another. iSeries is not for playing games. Linux is terrible when it comes to video editing. A secure Windows web server? Mmmmm, good luck administering that one!
Linux is a broad brush mate. There are some many different flavours/distros that you need to pick a task for which Linux is suited, then find the best distro to perform that task. Which pretty much means that you have to treat each distro as a separate OS. Again, you wouldn't use full blown Red Hat for a proxy/firewall box, would you? You would use a hardened secure distro such as SmoothWall. Hope that clears up that confusion for you...
Now, the experience question. I have been playing with PC's since I was a nipper. Our first computer was an IBM PC XT - with a Turbo card that clocked it to 7MHz from 4.77MHz. An extra 128Kb of RAM to take it to the magical 640Kb almost bankrupted my dad. He worked for a software house, hence the PC over a Speccy or Beeb B. Back then, all there really was in terms of PC games were text based adventurers such Zork or Hitchhiker's Guide, along with a few Atari ports and other keyboard breakers (as dad called them). Being a little concerned that his sons would indeed break the (probably expensive) keyboard, we weren't really allowed to do much gaming. Long story short, I borrowed "Learning IBM Basic by David Lien" from one of the programmers (there weren't developers back then) and read it cover to cover. Taught myself how to program and ended up doing Software Engineering at Uni. I have worked as a Field Service Engineer for a Third Party Maintenance Company, System Admin on a 200 user Windows network, System Admin on AS/400, Database Admin on AS/400, Developer and Systems Integrator on AS/400, System Admin on AiX boxes, System Admin on Sun boxes, System Admin on LAMP stacks of various sorts, Web Developer, Project Manager and now am a Systems Architect for a Super Major Oil Operator - that's one of the Big 5. I've been around the block mate, is what I'm trying to say. All of the above has been in a professional capacity, I have an avid interest in computing in general, so have a few little projects of my own on at the moment.
Anyway, background over. What all of this experience has taught me is that using the right tool for the job is paramount. All this buttering up and fanfare behind Linux is getting almost as bad as the Fanboi's from the land of milk and no reception. People need to have an informed opinion: not just an opinion. They're like arseholes, everybody's got one...
Oh, and if you are going to have a pop at someone, at least make sure you can spell and your grammar is correct...
That is all.
The title is required, and must contain letters and/or digits.
well actually I am running secure servers on windows, and I've also seen professionals editing videos on linux. So bad example. I guess every OS can be used for everything.
How did they discover the spyware. How did the spyware get onto the servers and what does it do. Is it generic spyware or specifically targeting Dell servers running Windows.
Type 1 Hypervisors in ROM
I'm just waiting for malware in a ROM'd (OK, probably Flash really) type 1 bare metal hypervisor to appear in the wild. I understand that there have been some proof-of-concept work (SubVirt and Blue Pill are two that come to mind), but if a way is found by undesirables to inject code into a Hypervisor in real environments, then it is going to be very ugly to take it off!
I'm also intrigued about how much the "black box" service processors and KVMs that are in everyone's network now can actually do, and whether they can be infected or even trusted in the first place.
The prospect of these really scares me.
Sorry, AC'ing this comment, as I don't want my bosses to see that I am thinking about these things.
I notice the Dell rep in the support forum states that it only affects replacement motherboards. Incidentally, that piece of information reveals how this issue doesn't affect new servers; read ALL the words. I am fairly sure that Dell refurbishes bad parts and uses them for warranty replacements. My guess is that someone at the refurb facility either goofed up royally or intentionally loaded something that is potentially malicious.
Dell needs to come right out with the details, because without knowing the full truth, people will speculate (as I have done) and likely will not feel confident purchasing Dell hardware. If I were one of the affected customers, I would absolutely demand a technical briefing of the scope of this vulnerability. Otherwise, how could I ensure data security?
The data farmer in the Dell
Even though I run Linux on my laptop I like Windows and will defend it until it dies horribly.
Don't be a hater guys.
Save that for the Apple Fanbois.
(I last saw this on a printer driver install from some (unnamed system starting with an H ) had to reinstall windows and gave said printer its freedom; kicked it well into the parking lot and injured my foot in the process.)
How often do you test new hardware for 'altered' architectures? As I recall, the US Navy was a bit concerned over sourcing for the computer internals on some of its fighting ships. Seems that hardware can be built to open doors. The ic's may be designed in secret labs, but where is it that budgets determine where chips are mass produced and put on to boards?
rom back doors
Hey Bob isn't this rather large for bios? Rick don't wory it's from Dell. Two months later the bank they work for gets hacked and their clients atm card gets cloned. The banks claim that chip and pin is perfect and thier customers are liars.
Time to update the article
Aliens, because everything is a conspiracy and they stole my coat!
More info from the forum page.
Here are further details regarding the instance of malware introduced on some service motherboards discussed on this forum that affects a very small set of customers. We are proactively contacting identified customers and are working with them to quickly resolve any potential exposure.
There are important pieces of information to note:
1. This issue does not affect any Dell PowerEdge servers shipped from our factories and is limited to a small number of the replacement motherboards only which were sent via Dell’s service and replacement process for four servers: PowerEdge R310, PowerEdge R410, PowerEdge R510 and PowerEdge T410. The maximum potential exposure is less than 1% of these server models.
2. Dell has removed all impacted motherboards from the service supply. New shipping replacement stock does not contain the malware.
3. The W32.Spybot worm was discovered in flash storage on the motherboard during Dell testing. The malware does not reside in the firmware.
4. All industry-standard antivirus programs on the market today have the ability to identify and prevent the code from infecting the customer’s operating system.
5. Systems running non-Microsoft Windows operating systems cannot be affected.
6. Systems with the iDRAC Express or iDRAC Enterprise card installed cannot be affected.
7. Remaining systems can only be exposed if the customer chooses to run an update to either Unified Server Configurator (USC) or 32-bit Diagnostics.
Dell takes customer security and privacy very seriously. Although we are not aware of any reports of customer related issues, we are proactively working with customers to resolve any potential exposure.
Concerned customers can contact Dell technical support at: US_EEC_escalations@dell.com
We will continue to update this forum as new information becomes available or questions arise. [/quote]
A bit of a concern, all this... Particularly for people like me who are currently in the middle of determining which Dell server to buy...
I think I might start looking at the IBM and HP options again ;-)
In a choice between Dell, IBM and HP I'd take Dell every time without hesitation.
To be fair Dell did come clean
from the last post it would suggest buying a new one is fine unless you get a repair
Not happy Dell let it get in, but I will credit them for coming clean quickly and updating
as news developed.
I imagine I'm not the only one here old enough to remember when one of the music vendors released a whole raft of music cds that had garden grade malware installed (the sort Norton and McAfee were detecting while sleepwalking, not the DRM crap that they've recently intentionally included) on their mass stamped CDs.
Respect to Dell for being open, other havn't about past issues like this
The whole area of added flash storage on a mobo is one obvious area of potentual infection to all, respect to Dell for being open and admiting it was there fault (or least part of there channel) instead of fobbing off over it and covering things up.
It's not the first time something like this has happened, indeed I've seen in the past modified BIOS's that had keyloggers, even some keyboard's. Hell even mice thesedays have a small computer with flash storage built in. Lets not also forget the utter trust of the firmware upon your HD's and to top it all the network cards that in servers tend to have more local processing power and flash storage than a mobile phone.
These are all area's were your AV's dosn't fully cater for if at all.
I might also add that the whole area of graphics cards or GPU's which have emense processing power, full access to the PCI bus and memory and plenty of local storage - How long until some AV firm starts selling AV for graphics cards or is the whole cload computing model realy invulnerable from maclicious code. No there not.
Bottom line anything that runs code that can be changed is a potentual area of exploitation. When we have an OS that at least checksums the firmware on all connected hardware, until then at the very least will we have a good reason to be paranoid.
But in this instance, Dell did the honourable and right thing and got proactive in respect to there customer base. How they found out we don't know, maybe a customer advised them and they reacted fast as or maybe this is a customer issue thats dragged on for age's with Dell failing to finaly pass the buck, we don't know. All we know is Dell had a problem and premtivly noticied customers. I'm sure they will also cover for any costs incured and damage done, but again, we just don't know.
Only thing I expect to come out of all this is hardware carrying malicious code seal of approvals with backup insurance to prove the point. Approved by Norton stickers anyone, whole logical sane area of income for the AV chaps to cash in on, and why not. I know the kit wont hurt the electricty network due to a CE sticker, why not a AV sticker saying this has been checked for virus's etc. Or do I need to be working in AV sales :o).
Thumbs up, there doing the right thing here from what we know and that has to be supported as we like being treated fair.
Sorry but I'm with Harry
I wouldn't bring up the Linux thing in the light of this being a flash-based problem, but the fact of the matter is that Harry's right and you guys are just being rude by calling him a fanboi and all that... no distro I've seen in years will mount any removeable device (which would include the onboard flash..) so that files are potentially executable. This particular vector would be entirely ineffective.
Anyway... if these are refurbished boards, I wonder if it was even introduced at Dell? Perhaps it was introduced on the site of the previous owner of the board, and then either spread at a Dell refurb plant, or didn't actually spread at all, just wasn't removed?
The one place we wan't bloatware.
In firmware. Flash enabled firmware should fill every available byte of storage with working code even if it's just padding initialisation and other low frequency modules with interspersed no-op instructions. Leave no room for malware.
As for how it happened? How did it happen with those digital photo frames? How does it happen with tradeshow give aways? How does it happen with CDs and DVDs?
Some twat dropped the malware into a master image somewhere and boards got infected.
More crap from Dull
Donut boy stikes again. Do you want a Krispy Kreem donut with that malware?
I'm waiting for the moment...
Won't take long now.
I'm mean, what consumer can resist the goodness of those features?
We need a Marketdroid icon. Saintly Bill will do.
I don't like Dell, but fair is fair.
Can't blame Dell too much on this, as this can happen from so many different angles it's not funny. Not even remotely.
At one place I worked at (before the internet days), they prevented malware from coming in by one simple rule. If you turn up on the premises with a personal disk (or other media type), you no longer work there. Period.
At another place, after determining our infection was due to one of our offsite offices giving the gift that keeps on giving (not mentioning any regions - asia).
Took us two days to clean up our end, and told them to clean theirs. After insisting they were not at fault, we disconnected them from the VPN till they cleaned up. Oh boy did they scream. But stopped once they DID admit to being the source of infection.
Apparently, their delay in cleaning up the mess was because it was impossible for them to rebuild any systems from scratch - every single program they had was pirated. And they no longer had the pirated software or keys, or keygens.
They failed to mention this because over there, it's normal. It gets worse, but under a REALLY tight fist, we manage to mostly keep things within reasonable bounds.
Can just see the tagline
"Now available.. With Free Malware!"
Epic fail from Dell again.
- Product round-up Coming clean: Ten cordless vacuum cleaners
- Worstall @ the Weekend BIG FAT Lies: Porky Pies about obesity
- 'Snoopers' Charter IS DEAD', Lib Dems claim as party waves through IP address-matching
- 'New Stuxnet': Government-grade SOFTWARE WEAPON 'Regin' described
- The next big thing in medical science: POO TRANSPLANTS