Virgin Media taps Microsoft in lengthy email outage Virgin Media customers have been suffering email outages for several days, prompting the firm to call in Microsoft engineers to help with an urgent upgrade. A mysterious configuration problem was identified on one of VM's eight email server clusters last Wednesday. Microsft engineers have struggled to identify the cause, …
My wife has been having trouble connecting to the email servers but we are NTL customers with @ntlworld accounts.
Either a conisidence that she has had problems since they started to have problems or they are not sure of the extent of the problem (the latter I suspect).
Still it is nice to see they have spent good money putting a cover on the Metro this morning to tell everyone on 4meg BB they they are now on 10meg BB rather that using the money to improve service.
Microsoft Exchange always seems to have given problems to Blueyonder, now Virgin, which is why I never use their email service. Personally I think Sky went down the right route by outsourcing it all to a company like Google - although the way the migration was handled obviously was pants.
Virgin should do a deal with Google / Hotmail / Yahoo or someone to run their email and concentrate on running the core network.
Go re-read the article
The current system isn't a MS Exchange, they are migrating to one.
The article is dosnt mention anything about there current setup.
And M$ have been called in to perform the migration as they cant get the current setup to work. I do love the fanboys who just read MS and jump to go on a bashing spree
/golfclap
Exchange is more than just an SMTP mail transport agent (MTA, as that is what SMTP defines). It's also a POP3 and IMAP server.
I'm not sure whether exim on it's own would do the job, and it is doubtful that it would scale to handle the large number of users that VM serves. You would probably need something like Dovecot, or the pop3 and imap features of Postfix.
I won't be on them much longer - moving to Hull (work) and the soddin monopoloy (even though they get arsey when you tell them it constitutes one) of Kingston Communications. I rely on gmail anyway.
Paris cos they say they aren't obstructing competition in the area, but by use of strange hardware and high wholesale price they are.
To all the MS fanboys jumping to defend them... why exactly?
Ok the article doesn't say in detail what they're migrating "from" but the comment that: "We have been working with our platform supplier Microsoft" and that they are now migrating to a "Microsoft Exchange 2003 platform" suggests that they are currently already on some form of Microsoft Platform to deliver their service (Exchange 2000 perhaps?) and are flailing about rebooting the systems, reinstalling software and now upgrading to Exchange 2003 (probably for free) to try and see if thats ready for servicing an ISP scale mail system. Anyone in the know care to dish the dirt?
"The current system isn't a MS Exchange, they are migrating to one.
The article is dosnt mention anything about there current setup."
If current system ($cs) is null (undefined), then I suppose $cs != exchange would be true. Outside of programming though, if the linux fanboys cannot make an assumption about the current system being exchange, then you cannot make an assumption about it not being exchange.
What the article does state is that they are accelerating the migration to exchange 2003. Maybe *their* problem is that they are moving to a 5 year old piece of software.
"Microsft engineers have struggled to identify the cause, forcing several reboots"
I can't believe that a company the size of Virgin Media is using Microsoft email server software.
I am a VM customer (ex Telewest) but fortunatly use my own e-mail server and as such have not been affected by the outages.
Anyway back to the point.
hmmm Linux./Unix/Solaris/BSD .... you know the operating systems that run the majority of the entire internet..... where you can stop/restart/start services and update non-kernel components without having to reboot.
Windows on the other hand :-
<MCSE Dave> Hey, Bob, this dang server still aint working right
<MCSE Bob> Have you tried rebooting it?
<MCSE Dave> Yeah 3 times already, but I have Vista at home, so I know that it's usually the 4th reboot that fixes it ... hangong a sec <CLICK> <CLACK>
<MCSE Bob> Hmmm that didn't fix it either. What else can we try?
<MCSE Dave> I'm just going to see if there are anymore updates we can install from Windows Update..... I don't think we have updated Media Player to the latest version yet.
<MCSE Bob> Yeah great, cos Windows update will force the computer to do AT LEAST 1 more reboot..... that should fix it.
<MSCE Dave> I hope so, we need to get this sorted and move onto the next job, we have a guy on the fith floor, who can't save his best scores on Freecell.
<MSCE Bob> Sounds like a nightmare, but nothing a couple of reboots won't cure.
Exim is *just* an SMTP server. It doesn't provide POP3 or IMAP (but that's what there are POP3 and IMAP servers for: it's better to be able to play one song from start to finish with no mistakes, than any number of cool riffs). It scales extremely well (just make sure you have plenty of RAM).
It also has a meaningful configuration file syntax. There really isn't much not to like about it -- unless you were educated at Oxford!
"Go re-read the article"
I did ..
"The current system isn't a MS Exchange, they are migrating to one"
If it isn't exchange then way did they ask MS engineers to identify the cause? Like why not ask the original suppliers?
'A mysterious configuration problem was identified on one of VM's eight email server clusters last Wednesday. Microsft engineers have struggled to identify the cause, forcing several reboots'
"The article is dosnt mention anything about there current setup"
It does, it says MS engineers are trying to identify the cause of the outages.
"I do love the fanboys who just read MS and jump to go on a bashing spree
I do love it when a ms.apologist refers to everyone else as a fanboy .. :)
Having just upgraded my phone & digital tv package to include broadband and v+ (which is nicer than Sky+) their email system is not very impressive. There doesn't appear to be any way to filter spam other than blocking the senders address! It's not a big problem as I already have more email accounts than I know what to do with. But, it does tarnish what is actually not a bad service - on the evidence of the last 10 days anyway.
I am one of their ex-blueyonder customers and the e-mail system is a mixture of Exchange and open source. There was a similar massive outage a few years back when they were entirely Exchange based, a few open source bits appeared to take some of the strain.
This is part of a header from an e-mail today:
Received: from smtp-in3.blueyonder.co.uk ([172.23.146.14]) by cluster2 with Microsoft SMTPSVC(5.0.2195.6713); Mon, 18 Feb 2008 15:44:14 +0000
Received: from eback01.blueyonder.co.uk ([195.188.53.212]) by smtp-in3.blueyonder.co.uk with Microsoft SMTPSVC(5.0.2195.6713); Mon, 18 Feb 2008 15:44:14 +0000
Received: from [172.23.170.141] (helo=anti-virus02-08) by eback01.blueyonder.co.uk with smtp (Exim 4.52) id 1JR8AA-0001DU-2U for --------@blueyonder.co.uk; Mon, 18 Feb 2008 15:44:14 +0000
Received: from [38.100.169.113] (helo=mta601.email.xbox.com) by exim13.blueyonder.co.uk with smtp (Exim 4.68)
We asked VirgTeleEuroBellWestMedia (as they are in this area, Eurobell box, Telewest endpoints, virgmedia branding) to move our cable service on the 27th February (We thought we'd give them a couple of days to mess it up a little before we actually moved). They said "Yes, that's no problem". They then went on to clarify. "We can disconnect you on the 27th, however we won't be able to connect you at the new place until the 29th". DOH!.
Still with how email and the service generally is right now with VM, we won't really notice a couple of days down time. It's not like it doesn't happen at least once a month as 'normal service'.
Icon, cos penguins are cute.
The lack of understanding around how enterprises work on here is astounding!
Also I know that the MS bods involved in something like this are not going to e idiots from the street! I have worked on large projects where senior MS engineers have been involved and most of these guys have a great understanding of the code behind the product and how it should work.
Before jumping to conclusions, you should perhaps take this into account. I'm no MS lover, but I keep an open mind.
That link in the article to the support ticket says what system it is - the affected cluster is Exchange 2000. Also they're not rebooting it - they're failing over between nodes, so it's individual servers that are going down, not the whole shebang. I wouldn't fancy migrating while it was in that state though.
Anyway, speaking of Exchange not being suitable for ISPs to run an email server - anyone know what Hotmail runs on?
As a long term customer I'm 99% sure they've been using Exchange for a very long time. At least since the days MS was a major shareholder in the old TW, at least for a short while.
I've got the mail headers to prove it and to the untrained eye it looks like they use Exim for mail processing but stick our mail boxes of Ex2k servers.
I've had intermittent and recurring problems accessing my Blueyonder mailboxes for a couple of months now. Because I use Linux and Thunderbird VM's technical support has said "sorry son, you're on your own because we don't know anything about Thunderbird. We know about Outlook." even though the issue appears to be with the mail servers; my TB is the latest version and is set up as per the recommended guidance.
Sometimes it connects, sometimes not. Of course, it was set up back when Telewest's support were based in this country and a very helpful chap in Liverpool confirmed I'd got everything right. Now you mention Linux to VM support and you might as well be talking Latin to them.
In the end I moved all my e-mail stuff to Googlemail. Problem solved.
Bottom line - from this user's perspective, VM failed. From the home user's perspective they just want reliable working e-mail. Something I didn't get so moved elsewhere. VM seem to lack a certain focus here.
Part of Virgin (The bit that used to be BlueYonder) use what used to be MCIS - Microsoft Commercial Internet Services - this was a service provider targetted package which provided SMTP, POP, IMAP, Authentication and NNTP in a single package.
I was involved in the first rollout of this pile of wotsit in the UK at Cable and Wireless back in 1998 - Microsoft ended up having to do 2 custom drops of SQL Server just to support the authentication needs of 50k customers.
Roll forward several years - Blueyonder are using MCIS for their email platform. MCIS doesn't have a very good reputation, but they blunder onwards with up to 20% mail loss as they go, but it sort of works.
Microsoft in time decide that MCIS isn't a good brand and rebrand it - it's now "Microsoft Hosted Exchange" which is effectively a tweaked version of Exchange itself scaled for ManyManyLots customers (BT Connect2Business had 1.1million users on it - don't know if they still have).
BT rolled Hosted Exchange 2003 a few years ago - by the sound of things Virgin are probably still on HE2000 and are forcing an upgrade.
Exim will scale to this size of user base, but it's only an SMTP server - BT Internet, prior to merging its mail services with Yahoo! was running Exim on its mail farm for 4 million users pushing something like 20 million mails a day. You still need a POP/IMAP server (BT used a customised version of Cyrus as I recall - it's been a while) and an authentication server, but LDAP servers are easy to get.
Thing is - if you want a *seriously* large messaging system you're really looking at what used to be Software.com's Openwave platform - it was designed to sit at the middle of a mobile phone network with tens of millions of mailboxes and punt messages whether they be SMS or SMTP.
Of course, the alternative is to craft your own - ask Yahoo!, Google et. al.
Instead of speculating....
[root@email ~]# telnet mailin.blueyonder.co.uk 25
Trying 195.188.53.99...
Connected to mailin.blueyonder.co.uk.
Escape character is '^]'.
220 exim8.blueyonder.co.uk ESMTP Exim 4.68 Mon, 18 Feb 2008 17:12:02 +0000
quit
221 exim8.blueyonder.co.uk closing connection
Connection closed by foreign host.
And :-
[root@email ~]# telnet smtpin.ntlworld.co.uk 25
Trying 81.103.221.10...
Connected to smtpin.ntlworld.co.uk.
Escape character is '^]'.
220 ESMTP server ready
214-This SMTP server is a part of the InterMail E-mail system. For
214- information about InterMail, please see http://www.openwave.com
They obviously have a mixture of technologies as any large ISP would.
"Anyway, speaking of Exchange not being suitable for ISPs to run an email server - anyone know what Hotmail runs on?"
Well... there was a big fanfare about them migrating off FreeBSD.... but when the Reg scooped that they'd only migrated the front end servers to IIS and the backend was still all FreeBSD it went all quiet. Perhaps they did migrate it all in the end, but I can't find a decent article about it anywhere. There's a tech-ed piece from 2000 describing the migration (http://technet.microsoft.com/en-us/library/bb496985.aspx)... but that predates the 2001 Register story about it just being the web front end that was moved (http://www.theregister.co.uk/2001/12/12/microsoft_hotmail_still_runs/).
So no idea what it's on. MS does however try to eat its own dogfood as much as possible (or just buy out someone elses) so I'd be surprised if they weren't using MS software by now.
Now you've hit a raw nerve. I have been a Virgin.Net customer for many years, and in general, have found the service fairly reliable (although this could be because I am not a cable customer, just ADSL), but their support STINKS.
I have a bare wire ADSL install to my own DSL router, to a Smoothwall firewall and then to a mixed wired and wireless network of Linux, MS Windows, Macs and even an AIX system (yes, this is at my home). I use a mixture of Webmail (for convienience) and fetchmail to pick up my mail from their mail server.
If I mention any one of these components in a support call, they just turn off and don't even bother to understand the questions.
I'm currently having capacity problems (I'm not getting even 10% of my paid-for 8Mb/s bandwidth), but am currently at the "have you turned off your computer and router and turned them back on" stage.
I took the rash action of sending them traceroute and ping timings to each of the hops, to show where the most likely bottleneck was, but I think I must have blown the recipients poor little mind, because I never heard anything back! They also took a very dismissive attitude when I complained that my allways-on link was being dropped several times a day, which screwed up my dynamic DNS entries. I now have to manually force my entry several times a day just so I have it working for some of the day.
One of these days, I will get so sick I will change, but I can mostly work without their help, so just get by.
My wife says that I should leave now. My coat is the one with all the network cable and micro-filters filling the pockets.
Oh dear, I'm really surprised they use Exchange for customers e-mail accounts. I would have thought they would have chosen a more robust Open Source solution, might have been fixed quicker too than relying on Microsoft.
I'm glad I host my e-mail with a decent hosting company.
Rob
P.S.
I wonder if they have to buy an Exchange CAL for each customer who has an e-mail account with them?
Not necessarily, they could be mail relays and no, ISP's don't use a mixture of technologies for the same thing, at least not when the whole thing is designed from the ground up. Thats what happens when you cobble several crap companies together into one big crap company through mergers and acquisitions.
Linux is an OS.
However, you make a damn good point; even a bog-standard Linux distribution will come complete with Apache2, Squid and various other server-specific items such as a SMTP mail transport agent and Pop3 and IMAP services.
I'm not really sure how the EU would view this....
The penguin, because they're from the Southern Hemisphere.
Incoming SMTP goes via the EXIM front servers, to the spam filtering servers, to the Exchange SMTP servers (mix of 2000 and 2003 at the moment)
To the person who can't work out how to configure the anti-spam - it's off the options part of the webmail system.
This is the only reason I personally use squirrelmail, the availability of plugins to control extras like spam settings
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
