back to article Linode: Back at last after ten days of hell

Linode reckons its long outage has come to an end, although its most-current message says there may be “intermittent” issues for users, mostly of its Atlanta facility. At the time of writing, the status of all services was listed as “operational”, except for Atlanta which still shows as “partial outage”. The company has been …

  1. James O'Shea

    I guess that if you're a customer in one of the blocked areas (which is most of the world...) you're outta luck...

    1. Scoular

      They were out of luck anyway weren't they?

    2. Anonymous Coward
      Anonymous Coward

      The rest of the world will just have to stop running botnet-friendly OSes if it wants to see our, umm, botnet-friendly PHP sites again.

      In the meantime, those who know what they're doing can use proxies, if it's that important to them. And I expect that anyone running 'important' sites on Linode - or DigitalOcean, Vultr, AWS, Azure, etc - will soon be adept at migrating to another VPS host when their current one falls over.

  2. Richard Boyce

    How much does it cost an attacker these days to launch a large-scale attack? What's the going rate per Gbit of DDoS?

    I'm guessing that the majority of major ISPs still allow outgoing traffic with spoofed IP addresses. If only politicians worked less to undermine security, and started mandating that ISPs install filters to block spoofed traffic instead of mandating that they spend money on filters to direct traffic to politicians.

    All a bit depressing, really.

    1. Anonymous Coward
      Anonymous Coward

      The basic problem seems to be some assumption that firewalls are supposed to keep the risky internet out of the trusted intranet. Defaults should be that outside traffic cannot appear to be from the inside and that inside traffic cannot appear to be from the outside.

      Since the ultimate defense against malfunctioning nodes is to simply unplug them, a protective firewall really needs to be able to prevent spoofed UDP traffic (apparently impossible to do with 1&1's "advanced" Cisco Firewalls).

    2. Pascal Monett Silver badge

      Re: How much does it cost an attacker these days to launch a large-scale attack?

      If I am not mistaken, the attacker has next to zero cost in this, since the attack, and the bandwidth cost, is handled by the zombie PCs that are part of the botnet. So the attacker only has to send the marching orders to the botnet, then sit back and watch the mayhem unfold.

      The only solution to this is the proper cleaning of the infected PCs and the education of the users. The first will remain difficult so long as bug-ridden Flash maintains its existence, despite efforts to bury it. The second will only bring fruit if the cluebat becomes an accepted education tool.

      Either that, or some other OS than Windows becomes the default on Internet-facing user computers, preferably one which is inherently more secure than Windows so that, even if the brainless dolt persists in clicking on everything, the OS itself will resist becoming part of a botnet.

      And once botnets are history, DoS attacks will be much more difficult to perform, and (I think) next to impossible to do anonymously.

      1. TeeCee Gold badge
        Mushroom

        Re: How much does it cost an attacker these days to launch a large-scale attack?

        Except, of course, that if the brainless dolt persists in allowing root/admin privilege to all and sundry when asked every OS is vulnerable. Anyone who knows slightly more than fuck-all about computers knows this, which puts you in with the dolts.

        The real question here is how do we produce an OS that completely denies all admin level privilege to the end-user, also lets them do what they want, where installation / upgrade / additional software is all controlled by professionals and its health is continually monitored by same? Also how do we do this in such a way as to avoid the flaming 'tards of the internet immediately killing it with rants about corporate/government/alien sponges control?

        1. allthecoolshortnamesweretaken

          Re: How much does it cost an attacker these days to launch a large-scale attack?

          "The real question here is how do we produce an OS that completely denies all admin level privilege to the end-user, also lets them do what they want, where installation / upgrade / additional software is all controlled by professionals and its health is continually monitored by same?"

          Part 3 of your suggestion (which I like) simply comes down to the willingness of the end user to pay for it. Part 1 and part 2 seem somewhat mutually exclusive.

          IIRC an article on El Reg from a couple of days before, Red Star OS pretty much nails part 1. Not so hot on part 2, though.

          Well, here'a a xkcd to cheer you up.

          1. I. Aproveofitspendingonspecificprojects

            Well, here'a an xkcd to cheer you up (FTFY)

            At least with Android and Apple have got mobile phones sewn up tight. And as it is linux based you don't have to worry- or pay.

        2. Anonymous Coward
          Anonymous Coward

          Re: How much does it cost an attacker these days to launch a large-scale attack?

          "...how do we produce an OS that completely denies all admin level privilege to the end-user, also lets them do what they want, where installation / upgrade / additional software is all controlled by professionals and its health is continually monitored by same? "

          Microsoft would have you believe that's Windows 10 with automatic mandatory updates and the Windows Store only for software you want to install (just to keep you "safe"). Be careful what you wish for...

          1. defiler

            Re: How much does it cost an attacker these days to launch a large-scale attack?

            Everyone is merrily pointing fingers at Windows. Once never had a Windows machine compromised beyond adware. On the other hand, my Centos box was broken into via SSH (non-standard port) over Christmas.

            I was out. Internet was sluggish when I came home. Found the problem and fixed it within a couple of hours. My XBMC user for accessing my movies had a shitty password and shell access. It's an easy thing to overlook.

            Fixed now, and fail2ban to go in. Not going anon because I'll live with my mistake and I guess I deserve some ire...

            1. defiler

              Re: How much does it cost an attacker these days to launch a large-scale attack?

              Oh, the attacks came from three different IPs on three different subnets, all in China.

            2. storner
              Boffin

              Re: How much does it cost an attacker these days to launch a large-scale attack?

              I'd recommend "PasswordAuthentication no" in sshd_config on ANY system, especially those that can be reached from the outside.

              Add Google authenticator for 2FA if you are on the paranoid side (like me).

        3. Anonymous Coward
          Anonymous Coward

          Re: How much does it cost an attacker these days to launch a large-scale attack?

          "The real question here is how do we produce an OS that completely denies all admin level privilege to the end-user, also lets them do what they want, where installation / upgrade / additional software is all controlled by professionals and its health is continually monitored by same?"

          Errr, maybe Z/OS ?

          Truth is, it'll never happen for consumer-level equipment because consumers need to "consume" to keep the money flowing to the producers, and anything that seriously impedes the consuming is a Bad Thing and must be stopped. Security can go straight to hell and hang when there's money to be made.

          maybe I'm a bit cynical...

          1. petrovich

            Re: How much does it cost an attacker these days to launch a large-scale attack?

            iOS is locked down pretty tight, while still allowing users to consume what they want (assuming they only want things approved by Apple and the content industry).

    3. patrickstar

      Consumer grade connections already do this on a wide enough scale that most botnets don't bother trying. Hell, don't you need to install a custom driver to do it on Windows anyways?

      Doesn't really help when there are a couple of million hosts pounding you and your gear has no O(1) way of blocking source addresses...

      On non-consumer grade connections, it breaks enough things that it's rarely(?) done.

      1. patrickstar

        PS. On Linux a lot of DDoS bots stem from (generally PHP) web site mass-compromise. I.e. no root, so - again - no spoofability.

        (stupid edit time limit...)

  3. a_yank_lurker

    Curious

    Curiosity, does anyone know why Linode would be DDoS target? Asking out of my ignorance and nothing else.

    1. Mark 85

      Re: Curious

      Probably for the same reason others have suffered DDoS attacks lately... ProtonMail, Janet, certain root servers, Eclipse and others. Possibly probes of defenses or something more sinister. Seems kinda' unlikely to me it's script kiddies. Only the attackers know at this point.

      1. Robert Helpmann??
        Childcatcher

        Re: Curious

        To further complicate matters, the attack might have been targeted at Linode itself or one of its customers via their hosted systems.

    2. Tony S

      Re: Curious

      I don't "know"; however, I "suspect" that this is an example of some group flexing their collective muscles to test out a number of processes. Who they are and where they come from is unclear.

      I also suspect that these are probing operations, designed to tested operational capacity of both sides. Specifically, their ability to conduct the attack and the capacity of companies and agencies to respond appropriately. We may see a few more of these, possibly in the not too distant future; and then possibly even a full scale attack.

      I somehow doubt that this is the work of script kiddies; it seems to be too focussed and determined for that. I'm betting that the more senior members of the security community are privately very worried indeed.

      1. Anonymous Coward
        Anonymous Coward

        Re: Curious

        'I'm betting that the more senior members of the security community are privately very worried indeed.'

        Nonsense. They will be rubbing their hands.

    3. Sirius Lee

      Re: Curious

      Probably because they can. I have a very small and inconsequential site and I block all access from IP addresses associated with China. If I don't, it is hit constantly with attempts to sign in - hundreds per second. If I block one IP address, the attack comes from another IP address. So they are all blocked. I block many Russian IP address ranges for the same reason. Attacks still come from Turkey, Brazil, Ukraine (mainly the east) but are less coordinated.

      Maybe the attacks from China are with a purpose. If the response of sites in the west is to block IP addresses associated with the middle kingdom the PRC doesn't have to do any censorship because we are doing it for them.

      1. allthecoolshortnamesweretaken

        Re: Curious

        @Sirius Lee:

        I think you're on to something. From that point of view the guys behind the DDOS kill more than just two birds with one stone, so to speak, and also achieve the most prized aspect of questionable operations - plausible deniability.

      2. monty75

        Re: Curious

        You might want to look at fail2ban. It'll dynamically firewall off any IPs that make more than a user-definable number of failed login attempts.

        1. Peter2 Silver badge

          Re: Curious

          You might want to look at fail2ban. It'll dynamically firewall off any IPs that make more than a user-definable number of failed login attempts.

          My personal experience is that attackers rarely use the same IP more than once. When I get port scanned or spammed my experience is that it's done by thousands of different IP's, all scanning a handful of IP's (sometimes even one to an IP!) and spamming appears to have largely gone the same way.

          With antispam, I have honeypots set up for a lot of email addresses and I rarely get more than a couple of emails from a single IP which hugely devalues IP blacklists. On the flipside, this does mean that any given site learns a huge number of IP's from botnet members though, so perhaps somebody needs to come up with a automatic system for looking up and emailing the abuse contacts responsible for the IP's to take advantage of this.

          1. Anonymous Coward
            Anonymous Coward

            Re: Curious

            My experience is that the hackers from PRC/Hong Kong do use the same IP addresses. We log them and when the number of attempts reaches an annoying level, we block.

            This is from my logs from Sunday for one of the machines. I've removed most of the lines underneath as there's hundreds and hundreds of attempts.

            sshd:

            Authentication Failures:

            root (61.147.103.115): 919 Time(s)

            root (40.117.46.116): 258 Time(s)

            root (103.226.184.218): 252 Time(s)

            and another

            sshd:

            Authentication Failures:

            root (61.147.103.115): 919 Time(s)

            root (202.106.211.99): 588 Time(s)

            root (40.117.46.116): 258 Time(s)

            and another

            sshd:

            Authentication Failures:

            root (182.131.21.69): 925 Time(s)

            root (61.147.103.115): 919 Time(s)

            unknown (182.131.21.69): 279 Time(s)

            root (40.117.46.116): 258 Time(s)

            and another

            sshd:

            Authentication Failures:

            root (222.186.34.203): 299 Time(s)

            root (40.117.46.116): 258 Time(s)

            root (103.226.184.218): 252 Time(s)

            I tend to ignore anything under 1,000 attempts in a week as there's so many different ones. I look at the networks and ban the whole IP range. The record was a single IP address doing something like 200,000 attempts in a week. After a while you can see the regular single IP addresses and then bang, block the IP range. 90% are China/HK.

            These few rules below help get rid of most attempts

            ufw status numbered

            Status: active

            To Action From

            -- ------ ----

            [ 1] Anywhere DENY IN 43.229.53.0/24

            [ 2] Anywhere DENY IN 218.87.111.0/24

            [ 3] Anywhere DENY IN 221.203.142.0/24

            [ 4] Anywhere DENY IN 113.195.145.70

            [ 5] Anywhere DENY IN 221.203.142.70

            [ 6] Anywhere DENY IN 218.87.109.60

            [ 7] Anywhere DENY IN 59.45.79.117

            [ 8] Anywhere DENY IN 186.46.185.234

            [ 9] Anywhere DENY IN 23.254.211.124

            [10] Anywhere DENY IN 59.47.0.152

            I might just block the whole of PRC off but was unclear what the impact on the firewall would be.

            1. Anonymous Coward
              Anonymous Coward

              Re: Curious

              You have sshd open to the Internet?!

        2. Anonymous Coward
          Anonymous Coward

          Re: Curious

          "You might want to look at fail2ban. It'll dynamically firewall off any IPs that make more than a user-definable number of failed login attempts."

          I do that with iptables rules. Unfortunately it doesn't deal well with attackers who hop ports on the same address. Still better than nothing, and I supplement it by dropping large blocks of IP addresses. The worst culprits are on addresses associated with Chinese government networks, despite official denials.

          1. I. Aproveofitspendingonspecificprojects

            Re: Curious

            The problem with the attacks is that if they are government agencies learning weaknesses of other countries the way airspaces are probed by potential enemy air-forces is that they won't unleash the full impact until they want to complete destruction.

            Agencies such as our own governments seem to think short term gains from allowing their own spy networks free access to our own computers (in the lucrative war against vegetable processors as well as the need to protect Hollywood and etcetera) is too much of a good thing.

    4. Anonymous Coward
      Anonymous Coward

      Re: Curious

      Context and analysis are missing from the article.

      If I come to the article cold, I need to know in the first paragraph who/what Linode are so I understand what the DoS is denying to their customers.

      And analysis so that I can be given some idea of 1) why someone might DoS them, 2) the outcome.

  4. Your alien overlord - fear me
    Linux

    On their front page they guarantee 99.9% uptime. I take it they now won't go down for another 10 years :-)

    1. Pascal Monett Silver badge

      Let's just assume they won't go down until the next attack.

      In other matters, I wonder if this region blocking is going to become a standard in security considerations. This is anti-ethical to the very notion of Internet, but if a company knows it only does business in a given area, maybe it makes sense to block all other regions.

      Or maybe global companies might start thinking about blocking the rest of the world for each of their regional installations. Microsoft Europe, for example, would block everything that is not Europe, Microsoft US would block everything that is not North America, Microsoft Asia would block everything that is not defined as Asian, etc. That might pull the rug under the feet of those worldwide botnet attacks somewhat, and the impact for the honest customers would probably not be all that significant.

      1. Mike Pellatt

        Region blocking is pretty much SOP for many organisations for SMTP traffic, and has been for years. This just extends it down to the IP level :-)

    2. 7layer

      He-he, good one! ;) But aren't they all guaranty 99.9?

  5. Anonymous Coward
    Anonymous Coward

    These attacks are via botnets but I have to wonder just why someone with a compromised computer does not notice the extra activity on their internet link and try to find out what is causing it.

    Maybe it is an awareness thing but I would think it would impair their web surfing to a large extent.

    1. defiler

      It does

      My wife: Facebook's slow. Oh I just switched off WiFi and it's fine again.

      In the other hand I noticed something awry and chose to investigate. Most people wouldn't bother or wouldn't know where to start. Router reset only helped for 5 minutes or so. Enough to make me think my router was on the blink.

      1. PeteA

        Re: It does

        Do you permit password-based logins (including challenge-response)? I'd personally advise very strongly against enabling anything other than PK-based authentication on boxen which are accessible via the interwebs; for extra paranoia, you might want to consider limiting yourself to ED25519 cryptography as the others are either getting a bit long-in-the-tooth (DSA, RSA) or may potentially have been compromised by the NSA (ECDSA - see https://blog.cloudflare.com/how-the-nsa-may-have-put-a-backdoor-in-rsas-cryptography-a-technical-primer/ for a readable description). I've not seen any examples of SSH compromise when password-based authentication is disabled, so would be very interested if yours did.

    2. Anonymous Coward
      Anonymous Coward

      I have to wonder just why someone with a compromised computer does not notice the extra activity on their internet link and try to find out what is causing it

      Because with a "well-designed" DDOS botnet with a sufficiently large number of compromised machines, the traffic from an individual machine will be negligible.

  6. nilfs2
    FAIL

    Where is the cloud DR strategy?

    The cloud provider has to be seen just like another datacenter, you need to have a DR site in case the main DC fails, if you plan to put a service on Linode you should also be planning to put a replicated copy on AWS, SoftLayer, Rackspace, or any other respectable cloud provider (please note the absence of Azure and Google after "respectable cloud provider").

    1. Adam 52 Silver badge

      Re: Where is the cloud DR strategy?

      What's your issue with Google and Microsoft?

      Both companies are privacy compromising data gluttons in other areas but their cloud offerings appear to be clean (at the moment, with some caveats about some Google services).

      1. nilfs2
        Headmaster

        Re: Where is the cloud DR strategy?

        @Adam 52

        Google Cloud Engine is still wearing diapers, buggy, lack of features and lots of downtime; Azure was erased from my "OK cloud vendor" list right away when I found out they block ALL ICMP traffic!, typical morons thinking that ICMP is just for running ping and do DDoS attacks, not knowing that ICMP is a critical part of networking traffic, who knows what other bizarre things like that they are doing, but that's enough proof for me to not trust their technical skills (or lack of interest to apply non intrusive security measures).

  7. Mark Allen

    how wide is the geo-block?

    Have they published how wide their geo-block is? I have a client with a website on Linode who does business with Azerbaijan. He is not going to be happy if the block is thrown too wide.

    1. I. Aproveofitspendingonspecificprojects

      Re: how wide is the geo-block?

      How close to the Bush Bloc and Colonial armies is thatbyjan?

      And much hassle do today's invaders have to put up with?

      On the other hand how much of a nuisance can Iran and isis be?

      But there again, I suppose all three agencies would like the troops speaking freely.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like