German web hosting and server provider 1&1 Internet is currently experiencing an outage, which started around 13.30 BST this afternoon, leaving customers without access to their websites. Various UK sources told The Register the servers were down, along with all communications into the provider's offices including sales, general …
not a problem anymore?
Not sure if I would be affected, but my email & domain is hosted by 1&1.co.uk and everything seems to be tickety-boo. Would they have been affected too?
According to their twitter feed
Due to a database-outage, in combination with a software error, we suffered from a faulty root server configuration...
Therefore since about 9am EST internet applications of 1&1 server customers aren’t reachable. Only 1&1 customers who use a ...
dedicated or virtual server are affected. The majority of customers with hosting packages (shared/Dual-Hosting/Cloud) ...
hosting packages (shared/Dual-Hosting/Cloud) are not affected. Our technicians are working on the problem under high pressure.
We have a dedicated server and there definitely was an issue. Another piece of insanity is that they insist on using twitter to post updates (and have to split messages over several tweets). You've got a dedicated status page, use it FFS.
and its back!
Nothing new with 1and1 (based in Germany by the way despite it looking like a UK operation.
My company's website is hosted with 1&1, as are my emails, and had no outages all afternoon. Tracking shows consistent hits all afternoon with no outages. Must have been a partial outage for them then...
My personal domain is with them too and I haven't noticed any issues.
A couple of points though...
- the helpful Status Page also shows it was last updated in August - I'd always prefer to see that sort of thing updated every few minutes, preferably with a history of status changes too so you can still see that there was an issue even if it has now been fixed.
- what I really want to know in a case like this is 'have I lost anything (i.e. content, incoming emails etc.)
Reason for outage
Spoke to an engineer in one of the data centres, he told me that there was an issue with a firewall configuration update and the network basically stopped at a node in France. They were rolling out backups of the old configuration so as to restore functionality.
It can't ben only a "firewall configuration problem".
After 2 hours, our server was again online.
Now, it is since more than 1 hour offline....
I am looking now for another company (it's not the first time i have problem with).
You get what you pay for.
Successful cowboy outfit
The update they posted to Twitter and to Facebook long after the problem first occurred was subsequently deleted once an angry mob of customers - presumably unable to get through on support lines, which weren't working either - started laying into them.
While their uptime has improved, their customer service has always been lower than whale shit, worsened by the fact you can't understand what their offshore slaves are saying as they parrot lines from the scripts on their screens. And in any case, during such outages it has been impossible to contact them anyway.
You just need to look at their naff advertising to realise 1&1 are now primarily a marketing company. A marketing company prunes negative customer feedback; a marketing company keeps its fingers crossed that any actual downtime is swiftly rectified so they have no need of tarnishing their status page with anything useful.
In fact, the 1&1 status page at status.oneandone.co.uk currently says:
All systems functional
Last updated at 04.08.2011
So much for German efficiency.
All Dedicated and Virtual Servers were missing
All of my sites hosed on my dedicated server were down from around 1:30 and all access to the machine, but since my email is processed by the 1&1 servers that was just slow.
The web based control panel for dedicated servers is still down, and without that you can't re-patch to a different IP address. Also the 'new' control panel is totally unusable from mobile devices so while out on the road it is now impossible to do anything via the control panel even if it is working :(
I am curious about how their new dual hosting product worked out, did it survive the downtime?
It mirrors your site in two locations.
Can any dual host users let us know?
Nora Batty's knickers!
The Twitter post is incorrect: cloud servers were affected. Mine was down for 2 hours this afternoon, then back up... now down again...
Shared hosting servers have stayed up for me at least
And still no response from 1&1 support
6 Dedicated Servers, 200+ sites down all day :-( no support....
We have more than 200+ sites on 6 dedicated servers. Have been on the phone trying to get an answer... no response... had clients calling and emailing all day.
Will have to move all our domains 2000+ and servers to a new hosting company. Any suggestions much appreciated.
Have a look at OVH.com
I administer webservers hosted at OVH.com and it's a real bargain. Just avoid like the plague the KimSufi offering. One of my customer had a cheap KimSufi server and was happy until the motherboard fried. An OVH technician replaced the motherboard but also reformatted the hard drive without notice! I tried to explain my customer that I never had such problem with my NON-kimsufi servers (you get the support that you pay for). My customer nevertheless decided to switch to a 1&1 server. Now, after several problems and this new outage, he wants to go back to OVH...
Re: Dual Hosting
I run several sites on 1&1. My Dual Hosting site is fine, as are two other shared hosting sites I run. Only the dedicated server site is down. (And it is down again as of about 15 minutes ago.)
I have been through a number of web hosts and 1&1 has been the best as far as uptime goes. I have found their support to be generally good as well, but communication with customers is a big problem for 1&1.
So, business as usual then? 1&1 providing a good service would be more newsworthy.
Down again as of 1915
As of 1930 BST, 1and1 is down again. It came back around 1700 and has now failed again. I'm off in the morning!
All our servers after being down for and hour and a half earlier are yet again offline! tried to call this time the phone's aren't dead like they were earlier but the wait time is 11mins + earlier the phone was going dead after being on hold for a few mins so i didn't bother waiting, it's the dissapointment i can stand!
You get what you pay for i suppose and these guys are cheaper than a blue and white stripey can of beans! which is about all they're worth!
Last time I renewed the dedicated server I looked for an alternative since they have never provided a service one would call 'acceptable'. The problem was finding a provider who ACTUALLY supplies unlimited bandwidth and does not hide other charges in the small print. I know what I am getting with 1and1 - crap service - but I've had several years with just the occasional hiccup. So until another provided comes up with the same service for the price I'll just live with the crap ... and duplicate everything on the servers here ready for when they simply loose the whole data complex :)
Getting seriously pigging annoying now
Thankfully worked enough earlier to get some bits done I needed, but now down again. In fairness, I don't think it was ever officially "back up" again, but still... Another grovelling email about to go out to my suffering clients. Once upon a time the server support was pretty direct and decent, but now they're pushing a zillion and one virtual options, they seem to have had to remove that contact.
Cost me a packet
I don't even want to think about how much money I have lost today because of this. The issue is limited to dedicated servers and virtual dedicated servers. We have a dedicated server and lost everything at around 3pm. Right now it just came up but is very intermittent. I imagine the financial loss could be as much as €3000
I think you'll find that a couple of hours downtime is well within your the service level agreement. If that's the case and you can't afford that then you need to colocate and manage your own DNS.
We have a server with 1&1 that was down for several hours this afternoon cutting off the website and e-mail. But that was the first time in 8 years. I think we can live with that.
Really? You earn around €1000/ hour?Hmm, that’s around €8.7 million /year. And you are using only 1 dedicated server? With one provider only?
What are your contingency plans? Can I suggest you maybe think about spending some of that on improving your infrastructure. You can probably afford it.
Sorry but you deserve it
Unless you did a cost benefit analysis beforehand and decided 1&1 were worth the risk I have no sympathy for you.
If you know you have that much value attached to services from a company that is known for being crap you deserve all you get.
I suggest you actually start paying for a service commensurate for the value at risk if it all goes wrong.
Managing your own DNS and colocating doesn't help when it's the network itself that's been screwed up. All through the outage my server was "up" 100% of the time - you just couldn't get any data in our out of it.
2.5 hours outage and still nothing. Costing us a packet. With Xmas round the corner, might just consider running but I guess that won't be easy considering their lack of customer service.
Back at the time of writing, but whether it'll last is anyone's guess...
Giving their server monitoring a try, although not entirely clear where it monitors from (in terms of being useful) at least it did acknowledge the evening outage correctly.
The problem today appeared to be some sort of network routing issue - which is what seems to fail in the majority of server outages I have seen from 1&1 and elsewhere. I don't think that their server monitoring (which is more restrictive than they describe - it does not apply to managed servers) would help you here.
There are two monitoring services I have used with success. websitepulse.com and siteuptime.com
Oh, as for dual hosting, my understanding is that it is two different servers in the same facility, not at different facilities. Helps with disk and server crashes, not so much with network or power issues.
Their monitoring DID help in this instance (as mentioned above), but obviously not the perfect solution without it being properly off-sited.
careful with their monitoring service, from what I've read the notifications cost. Seems strange that they charge you to inform you that the service you pay them for isn't working
Hmm, I doubt that. The only option was an email notification (which worked impeccably with yesterday evening's outage). Of course, if you forward that to their SMS gateway there's a cost, but that would go for any email-to-SMS conversion they offer, not just in connection with server monitoring.
Up again after a reboot.
We have a dedicated server there, when I called about 5:30 pm CDT they still had their voice message saying all dedicated servers were down. I waited on hold until I got a tech person who said they had 'just finished' some updates but that I would have to reboot my server to reactivate it.
Once I rebooted it seemed to all come up ok. So anyone else out there affected you may need to reboot.
Facebook? Twitter? WTF?
I don't know about you, but I can't access either Facebook or Twitter while at work (certainly not while my server is inaccessible and the possibility of tunnelling goes out the window, anyway).
It didn't even occur to me to look on either of those sites for status updates. What are they doing putting valuable information about downtime on them, anyway? (whilst neglecting their own status page)
I'm personally planning on scaling back my usage of 1and1 and maybe switching from dedicated to "cloud" server. Anyone have any experience with it? I've heard that disk IO might not be good enough for running a set of DB-backed sites.
Our web forum (we rent a physical server from 1and1) was down for about 2 hours in the afternoon and then another 2 hours or so in the evening. Not happy.
It was very obviously a network issue as (a) many other 1and1 servers stayed available and (b) one of our other admins, who had one of the working 1and1 servers, was able to tunnel into ours using the local serial port and confirm that the server itself was fine.
The usual quandary in these situations is whether we change the DNS and put up a static web page advising people of the problem, or just wait for it to recover. In the end we chose the second option.
I've used 1and1 for 8 years, running two dedicated servers. This is the first and only time such an outage has occurred to my knowledge. Roughly 3 hours out of 70,000 hours. That's an uptime of 99.996% (Actually, rounding up of course that should be 100%) For the price that 1and1 charge, that's very impressive.
Oh, and I've had a dealings with 1and1 tech support that have been superb. Helpful, solved my problems, and quickly.
The only gripe I have with them is their lamentable communication yesterday when things were wrong. I see they have two 'social network specialists'. They were a waste of time - just when they were most needed! And that status page notice was also a joke.
So, just a bit of perspective please.
Let me know if you can find an equivalent provider for cost, support and uptime and I might consider switching, but to be honest, the hassle involved would far outweigh any benefit.
...it was down again this morning.
The great game of astrowars (astrowars.com) is housed by these cretins and the outage caused some issues...
15:31:53 - Sep 26 Your defending fleet was defeated by [RTT] akirabzh at Schedir 3. You killed about 55%.
We use ippatrol and they picked up and reported both outages quickly and accurately alerting us via email and sms.
Major Routing Problems
1&1 certainly wobbled badly yesterday.
You can of course make comments about getting the standard of service you pay for. But I'd mirror the comment from "Both Sides Now" that this is the first outage of this scale that I have seen in something like eight years and 100 server-years.
Another much smaller hosting provider that we deal with has been falling about today, because of "internal routing problems". Their non-technical people are telling me these problems were not related to any kind of planned upgrade (implying their network just spontaneously fell apart and is now taking hours to put back together. Hmmm....).
So is it just a coincidence that the two hosting providers have had really major routing problems on consecutive days?
Or is there some kind of common factor? Perhaps an as-yet-undisclosed vulnerability that is either being exploited, or is triggering emergency upgrades that are proving problematical?
- Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
- FOUR DAYS: That's how long it took to crack Galaxy S5 fingerscanner
- Did a date calculation bug just cost hard-up Co-op Bank £110m?
- Feast your PUNY eyes on highest resolution phone display EVER
- Wall St's DROOLING as Twitter GULPS DOWN analytics firm Gnip