back to article Today the web was broken by countless hacked devices – your 60-second summary

Today a vast army of hijacked internet-connected devices – from security cameras and video recorders to home routers – turned on their owners and broke a big chunk of the web. Compromised machines, following orders from as-yet unknown masterminds, threw massive amounts of junk traffic at servers operated by US-based Dyn, which …

  1. Doctor Syntax Silver badge

    Maybe..

    ..just maybe this will finally spur TPTB into taking some action.

    For a start oblige the manufacturers of IoTs to stop selling vulnerable devices until they're fixed.

    At the same time, put out a recall for all those currently installed to be upgraded - or do over the net upgrades if for kit that supports that.

    And then make it illegal to run a vulnerable device if it's connected to the net.

    The second item might well cost vendors more than the profit they made in the first place - good, it's time vendors were exposed to the costs of cutting corners.

    1. Mark 85 Silver badge

      Re: Maybe..

      Nice thought, but I think most manufacturers will just shut down the product line rather than do fixes. Profit and all that. As for "illegal".. that part would be ignored as any fines will be relatively miniscule and that's only if a law can get past the corporate lobbyists.

      1. Doctor Syntax Silver badge

        Re: Maybe..

        'As for "illegal".. that part would be ignored as any fines will be relatively miniscule and that's only if a law can get past the corporate lobbyists.'

        Fines can be whatever legislation and the courts make them. There's also the possibility of raising sanctions against ISPs who continue to permit their customers to continue to use such devices.

        As to lobbying, recent events have resulted in some large corporations having incentives to lobby for action.

        In general history shows that eventually potentially bad stuff does get regulated but unfortunately governments traditionally don't operate at internet speed.

        1. Destroy All Monsters Silver badge

          Re: Maybe..

          governments traditionally don't operate at internet speed

          Unless it is to exploit a moral panic to increase control in unsustainable ways for no good reason expect that "something must be done".

      2. pbryant

        Re: Maybe..

        "...only if a law can get past the corporate lobbyists." and the Republican Party.

      3. Tomato42 Silver badge
        Unhappy

        Re: Maybe..

        > Nice thought, but I think most manufacturers will just shut down the product line rather than do fixes.

        and nothing of value will be lost

        1. Metrognome

          Re: Maybe..

          What unadulterated bollocks.

          How do you outlaw the Chinese makers that flood ebay, gearbest, aliexpress and the like?

          Do you guys think that some army of standards enforcers will land in China and start shutting factories down?

          The Chinese manufacturers neither know nor care about these things; mostly the same stands for their customers.

          1. John Brown (no body) Silver badge

            Re: Maybe..

            "The Chinese manufacturers neither know nor care about these things; mostly the same stands for their customers."

            <tinfoil hat mode>

            Or, just maybe, it's all part of "The Plot"

            </tinfoil hat mode>

          2. Doctor Syntax Silver badge

            Re: Maybe..

            "mostly the same stands for their customers."

            It's the customer end that you start with. Does the kit meet UL/CE standards? If not then it becomes illegal to put it on the 'net in the relevant country or, even better, it becomes illegal for the ISPs to route it. It also becomes illegal to offer it for sale so if it's on sale from a local vendor then they get a visit from Trading Standards or whatever in that particular jurisdiction. If it's being offered for sale on eBay from China or wherever then eBay gets a visit.

            The manufacturers will get the message without direct action - they want to sell stuff, they meet the standards.

            Make no mistake, something will be done, the only questions are what and when.

            1. VulcanV5

              Re: Maybe..

              " It also becomes illegal to offer it for sale so if it's on sale from a local vendor then they get a visit from Trading Standards or whatever in that particular jurisdiction. If it's being offered for sale on eBay from China or wherever then eBay gets a visit."

              In the UK, Local Authorities run Trading Standards departments. Also in the UK, central government (i.e., taxpayer) funding of Local Authorities dwindles year on year -- as do the number of staff employed as Trading Standards officers. Quite how this ever-diminishing number of consumer protection specialists is meant to visit every vendor of unsafe cheap Chinese tat, whether sold on a real-world market stall in hundreds of towns throughout the country, or the virtual auction house of eBay, is beyond me. Using Denial of Commonsense as an approach to the issue of Denial of Service ain't going to help at all.

    2. a_yank_lurker Silver badge

      Re: Maybe..

      I doubt any legislative action will actually be all that effective. The average Congress critter is not noted for critical thinking skills but emotional pandering.

      Security is hard to do even when users are reasonably proactive. To many IoT devices ignore proper security because they make it difficult to update the device even for proactive users. This could be fixed, possibly without any new legislation. Use the existing defective product recall laws on the books since these are defective devices. After a certain period of time and genuine effort then nail the manufacturers with fines for selling and refusing to fix defective products.

      1. Ole Juul Silver badge

        Re: Maybe..

        A class action law suit by users of these devices would cover older models just fine. My non-lawyer thinking suggests that being put at risk without any warning labels would make a case. I want to see these socially irresponsible companies put out of business. I'm sure there are others willing and able to take their place.

        1. Mage Silver badge

          Re: Maybe..

          Problem is proving that the USERS/Owners suffered at all.

          1. John Lilburne Silver badge

            Re: Maybe..

            'Problem is proving that the USERS/Owners suffered at all.'

            Apparently it took down GitHub, Twitter, Reddit, Netflix, AirBnb so the world actually got smarter.

          2. Doctor Syntax Silver badge

            Re: Maybe..

            "Problem is proving that the USERS/Owners suffered at all."

            No It's the suffering that users/owners are causing to others that's the problem.

      2. Doctor Syntax Silver badge

        Re: Maybe..

        "I doubt any legislative action will actually be all that effective. The average Congress critter is not noted for critical thinking skills but emotional pandering."

        I think a few large corporations being exposed to risk like this will be able to apply as much emotional pressure as is needed to produce results.

      3. herman Silver badge

        Re: Maybe..

        I think the pretty useless FCC and CE certification standards should be expanded to include security standards and pen tests for connected devices. That will exclude the craprouter manufacturers from most of the world markets unless they improve their toys.

    3. macjules Silver badge
      Black Helicopters

      Re: Maybe..

      The "TPTB" would not take the action you require simply because Twitter and Netflix were down for a while. No, you need a DDOS attack on a bank, a hospital network, an ATC centre or anything that can seriously scare them.

      1. Doctor Syntax Silver badge

        Re: Maybe..

        The "TPTB" would not take the action you require simply because Twitter and Netflix were down for a while.

        Can't Neflix and Twitter afford to buy a few politicians do any lobbying?

    4. YetAnotherLocksmith

      But it was secure yesterday

      n/t

    5. Mage Silver badge

      Re: Maybe..

      There is actually no solution to this.

      1. Anonymous Coward
        Anonymous Coward

        Re: Maybe..

        There is actually no solution to this

        You could be right, but I think that this will spur the rise of a closed "internets" owned by Farcebook and Google. They can apply controls on these kinds of bots as well as controlling free speech. Dytopian future draws nearer.

      2. bboyes

        Re: Maybe..

        The devious cracker break-in technique? "...logging into devices using their default, factory-set passwords". Something comes to mind along the lines of "you can lead a horse to water..."

    6. heyrick Silver badge
      Stop

      Re: Maybe..

      "And then make it illegal to run a vulnerable device if it's connected to the net."

      Another fine law to make criminals out of ordinary people.

      I have an IPCAM. I wanted it mostly as a toy, but it is useful for keeping an eye on things when I'm not around. See what the cat is up to, etc.

      Out of the box, it uses uPNP to punch a hole in the router for itself. It announces its presence to several foreign servers, and it has a default telnet login of root/123456.

      I've hacked the startup script (luckily writeable) to replace the hosts file numerous times at boot to direct all of the domains that the camera uses to localhost (obtained by connecting the camera to network sharing on my PC and wiresharking what happened during boot). The uPNP failed as I've disabled that on the router. There's a STUN to an IP address that I can't do anything about (my router is an Orange Livebox so it doesn't do fancy things like blocking individual IP addresses). The default password cannot be changed. I can use chpasswd but the next time the thing is rebooted, the firmware writes a new passwd file with the root/123456 combination. I also very much doubt the online firmware upgrade is in any way secure. I will, some day, make a binary hack to the main program file to replace the firmware cgi filename with gibberish (to disabled that) and change the baked in password to something else. I tried a sleep 60 in the boot script, but the thing overwrote it with the default. It's of lower importance as you'd need to be in my local network to access it.

      I'm a nerd. I could play with this and fiddle with it. I'm sure many people will just buy the device, plug it in, and expect it to work with "the app". If that's all it takes to be a criminal, there's no hope.

      1. Wayland Bronze badge

        Re: Maybe..

        "Another fine law to make criminals out of ordinary people."

        It would be illegal to hack into someones network and spy on them. It ought to be illegal to create a Trogan program to do that. Is it illegal to sell a device like an IP cam that does that?

        There is a IP cam with a web interface that Google has spidered into it's search. You can find them and view the video. You could probably also upload new firmware to someone elses camera. They did this to UBIQUITI wireless kit earlier this year and those things had passwords.

      2. Mage Silver badge

        Re: Maybe..

        " it uses uPNP to punch a hole in the router for itself. It announces its presence to several foreign servers, and it has a default telnet login of root/123456.

        I've hacked the startup script (luckily writeable) to replace the hosts file "

        Disable uPNP on your firewall / router.

        Setup a VPN (properly) to your home network if you want to remotely access stuff on it.

        1. heyrick Silver badge

          Re: Maybe..

          "Disable uPNP on your firewall / router."

          That was the second thing I did (after changing the router's default password). I spotted the uPNP requests in wireshark. As for uPNP itself - horrendous idea. Anything that needs to receive incoming data can fail nicely and/or ask for permission.

          But letting IoT devices grant themselves authorisations? Ain't gonna happen.

          [Bootnote: Orange sets the Livebox do support uPNP by default. People can buy stuff, plug it in, and "it just works". I wonder how many even understand what this process entails?]

          1. Steve Davies 3 Silver badge

            Re: Maybe..

            Stuff the routers/firewalls supplied by the ISP's.

            Make your own Firewall box that sits between the ISP router and your network devices. Then you can control everything and these crap devices can't get out and create links to the mothership.

            Also make them on a separate subnet to your printers and computers and you know, good stuff.

            None of these devices will get on my network even though I already have my own Firewall made from a fanless NUK.

            We need to make the stores and online tat shops like Amazon and Ebay stop selling this crap. Only then might we get somewhere before it is too late.

            Getting the politicians to act before we loose a country from the internet for say a week will be impossible I'm sad to say but we the more informed amongst us can do our bit and make sure that we are not part of the problem.

            1. John Brown (no body) Silver badge

              Re: Maybe..

              "We need to make the stores and online tat shops like Amazon and Ebay stop selling this crap."

              Since both Amazon and EBay were affected by this outage, one wonders if either or both of them will take any notice. Did it hit their bottom line in sales? Chances are, no, it didn't. Sales may have dropped short term but most people trying to buy will simply try again later, so over all, the bottom line was barely touched, if at all.

              Now, if we can get some non-thinking US Congress Critter to jump on a band wagon and scream from the rafters that the US economy lost $billions in trade because of this....

      3. Richard Simpson

        Re: Maybe..

        Well maybe it would be excessive to actually prosecute end users, but running insecure devices could be made illegal indirectly via ISPs. I think it would be perfectly reasonable for ISPs to be required to identify customers whose devices are part of these botnets and then warn those customers. With the legal stick being that if the customer doesn't fix or disconnect the offending device in a reasonable period (say a couple of months) then they get cut off until they do.

        1. heyrick Silver badge

          Re: Maybe..

          "With the legal stick being that if the customer doesn't fix or disconnect the offending device in a reasonable period (say a couple of months) then they get cut off until they do."

          Aaaaand.... how long until somebody goes running to their lawyer because the compromise that did the damage in the first place came from.... yup, you guessed it. The Internet. Provided by the same ISP now making "fix it or else" threats.

      4. Doctor Syntax Silver badge

        Re: Maybe..

        "Another fine law to make criminals out of ordinary people."

        Dunno where you live but hereabouts if you're running an unsafe car on the roads you can get a conviction however ordinary you are.

        1. DropBear Silver badge
          WTF?

          Re: Maybe..

          "Dunno where you live but hereabouts if you're running an unsafe car on the roads you can get a conviction however ordinary you are."

          And cars have garages than can grant MOTs. What non-God-Tier-Entity do you have in mind that can in good faith assert that a given device is "safe"? It's exceedingly rare to discover major faults in an existing car which is why recalls work at all; with computing, it's the daily norm. So do please tell me you intend to equate "safe" with "all patches issued as of today being applied" so I can laugh all next week.

        2. Gio Ciampa

          Re: Maybe..

          Knowingly, yes...

          ...but I'll wager that 99.9% of the compromised device owners even knew they were involved.

          (I await the botnet running on (mandated) "smart" energy meters with interest...)

          1. heyrick Silver badge

            Re: Maybe..

            "(I await the botnet running on (mandated) "smart" energy meters with interest...)"

            Here in France there is a somewhat hated new smart meter called "Linky". It is not legal to refuse to accept it, and if you persist then EDF will back down and just bill €€€€s call out charge for each time the meter is read.

            I don't know how it talks to the mothership, but it'll be interesting if they think it is going to talk to my wifi. I can use my crappy IP camera as a good reason to say "either I audit the source code of this thing or you find some other method of communication".

            As an aside - a newspaper article quotes EDF as saying that the Linky does not catch fire. It's just incorrectly installed. Wait, remind me, exactly who installs meters? I also await with interest the first time this thing gets hit with lightning. We have overhead three phase to the house. It gets directly hit once every two or three years, and proximity hit several times a year. Our old meter predates me but takes this stuff in its stride. Is it optimistic or just silly to expect the Linky to be as reliable? What's worse - if there is a really bad storm, I can throw the breakers and turn everything off. Well, you can't take the meter out of circuit. Hmm.

        3. Kernel

          Re: Maybe..

          'Dunno where you live but hereabouts if you're running an unsafe car on the roads you can get a conviction however ordinary you are.'

          There's a difference between running a car on the roads that you have knowingly allowed to become unsafe, as opposed to one that was manufactured unsafe but you bought on the not unreasonable assumption that the manufacturer knew their business.

          There's always some dick-wit who tries to compare to cars, isn't there?

      5. AndrewDu

        Re: Maybe..

        "The default password cannot be changed"

        Dear God.

        It's almost like the manufacturers (or somebody...) wanted that device to be insecure and remotely compromisable.

        OK, I'll take off my tinfoil hat now.

    7. The Man Who Fell To Earth Silver badge
      FAIL

      Re: Maybe..

      Maybe some smart developer should make a free tool so that people can at least check out their local network for compromised devices.

      Not me.

      I'm too busy: https://www.youtube.com/watch?v=VASywEuqFd8

    8. Planty Bronze badge
      FAIL

      Re: Maybe..

      This is obvious clickbait, it suggests all IOT devices are vulnerable,biy the reality is, its a single manufacturer (XiongMai Technologies) that had a default password and login.

      Worse still, this company only makes IP cameras, so to suggest this ddos was caused by routers, thermostat and toasters is just pure clickbait Horsecrap.

      It's however fashionable this month to hate anything IOT, so let's just ignore that....

      1. Doctor Syntax Silver badge

        Re: Maybe..

        "This is obvious clickbait, it suggests all IOT devices are vulnerable,biy the reality is, its a single manufacturer (XiongMai Technologies) that had a default password and login."

        The answer lies somewhere in between. It might be a single manufacturer in this case and not everything is necessarily vulnerable but there have been enough reports of routers with telnet ports open on the internet side etc. You don't need to look back very far in el Reg to pick up these.

      2. Stoneshop Silver badge
        Headmaster

        Re: Maybe..

        the reality is, its a single manufacturer (XiongMai Technologies) that had a default password and login.

        That's a definition of 'reality' of which I was not previously aware.

        The Mirai code contains a list of default username/password combos for a number of devices of varying functionality, not just IP cams.

    9. Kiwi Silver badge
      Linux

      Re: Maybe..

      And then make it illegal to run a vulnerable device if it's connected to the net.

      That'd mean kicking all them Windows users off the net.

      And all Macs.

      <unreadably small font>And for that matter, my Linux machines probably have some vulnerabilities in them that haven't been discovered... Yet... </unreadably small font>

      Seriously though.. That would kill all sorts of development work. Who would write and test code knowing that if they didn't find a security flaw, they or their customers could end up having to pay some sort of fine or worse? I hate Windows insecure crap as much as anyone, but there has to be some limits in here..

      (really must stop posting at 3am too..)

    10. Anonymous Coward
      Anonymous Coward

      The horse is already out of the barn and the barn's burned down

      But you blokes want a law to "fix it"

      Brilliant! Ain't no law gonna fix this problem. Massive bot armies are rampaging.

      We'll need a technical solution that ignores their requests. We'll basically have to turn them into millions of dead devices.

      And let the class action lawsuits on behalf of the consumers proceed at that point.

  2. Glenn 6

    Standards Bodies need notice

    In North America, you can't sell your electronic wares unless you have either a Canadian Standards Association (CSA), United Laboratories (UL), and possibly Federal Communications Commission (FCC) certifications to make sure they meet certain quality, safety, and in the case of FCC, RF emission standards.

    Perhaps it's time those bodies also include network safety standards being met? Companies need to be held to a high standard on these things, and they're clearly not.

    At the minimum, when things like this happen, there needs to be an investigation, and laws in place where corporations who cheap out on proper locking down of their devices are held to account.

    1. Doctor Syntax Silver badge

      Re: Standards Bodies need notice

      "Perhaps it's time those bodies also include network safety standards being met? Companies need to be held to a high standard on these things, and they're clearly not."

      Agreed. This is something I've been saying for some time. Also it should be added to CE requirements in Europe.

      The trouble is the existing deployed fleet. Those need to be fixed or taken off-line if they're not fixable.

      1. Dan 55 Silver badge

        Re: Standards Bodies need notice

        CE requirements would be useless, CE is crap, it's the manufacturer which self-certifies.

        The manufacturer should pay for tests by an independent body before going to market. No pass or no testing means fines for the manufacturer if they bring it to market and fines for the retailer who stocks it.

        Yes, this will drive up the overall price of goods, but, guess what, security costs.

        (I did say a few days ago that a 'not certified' sticker would warn the customer not to buy the tat and choose some tat with 'certified' sticker instead, but in the light of recent events that obviously isn't going to work.)

        1. Mark 110 Silver badge

          Re: Standards Bodies need notice

          Just enforcing a standard that all devices need a unique admin password of certain length structure and randomness ought to be a good start and not that hard for a device manufacturer to implement.

          1. JLV Silver badge
            Paris Hilton

            Re: Standards Bodies need notice

            Nice. But what about a default, one-time use, std user/pass combo that you _need_ to change on setup.

            Hardcoded into default factory setting, but that can only be done from a physical switch. Higher price point devices can implement other solutions for when remote password resets is needed.

            Basically demonstrate that you've spent at least 10 mins around a beer thinking about security. This may yet be a wakeup call.

            Plus, imposing reasonable import regulations re being fit for purpose should please all the nationalist types, no?

            Paris cuz she's better at security than some of these folk.

        2. DainB Bronze badge

          Re: Standards Bodies need notice

          So you will have strict requirements in EU and USA and will be attacked by botnet of routers from South America and Asia. How exactly your idea going to stop that ?

          1. Stoneshop Silver badge

            Re: Standards Bodies need notice

            So you will have strict requirements in EU and USA and will be attacked by botnet of routers from South America and Asia. How exactly your idea going to stop that ?

            "We can't stop them all so we might as well do nothing".

            1. Charles 9 Silver badge

              Re: Standards Bodies need notice

              ""We can't stop them all so we might as well do nothing"."

              In this case, it's accurate. It's not worth swatting one angry bee because there are a million more after you. You really DO need an "all or nothing" solution to it or the ones that slip by kill you.

              Problem is, sovereignty gets in the way. How can you regulate devices when they can just be shipped direct from companies who don't care?

              1. Stoneshop Silver badge

                Re: Standards Bodies need notice

                Problem is, sovereignty gets in the way. How can you regulate devices when they can just be shipped direct from companies who don't care?

                When I try to buy a laser device from Alidealgoodbest, I get a notice that "due to regulations, we can't sell lasers over $smallnum mW to @countries", probably followed by "Kthxbye" if that laser is over $smallnum mW and I'm in one of @countries. So that part of international regulation enforcement works, more or less, and I don't see why it can't be extended to cruddy IoT stuff*. There's also your country's customs between China and you, and while your individual shipment may or may not get caught, a container full of uncertified idIoT tat is unlikely to reach $shadydealer.

                * once appropriate regulation is in force, which will quite likely take a while.

                1. DainB Bronze badge

                  Re: Standards Bodies need notice

                  You're trolling right ?

                  You need to ship laser device by mail. You don't need ship IoT device to any particular country to cause damage, same device in Venezuela will do as much damage as if it was in California.

                  1. Stoneshop Silver badge
                    FAIL

                    Re: Standards Bodies need notice

                    You're trolling right ?

                    I will not buy this record, it is scratched.

                    Look, if you want to wallow in your opinion that any action is futile, go right ahead. I happen to disagree.

                    1. DainB Bronze badge

                      Re: Standards Bodies need notice

                      Well, you're either trolling or don't have any idea what you're talking about, it's up to you.

                      I don't agree with you on concept of Internet of trust where only allowed devices can access it as implications of that are too far fetched.

                      First of all, it's not possible to implement simply because there's no concept of global standards and what you're talking about would not work without it, Internet traffic does not care about your regulations.

                      Second - who and how decides which device should be banned ? Again see #1, banned in one country does not mean much in other.

                      TLDR There is solution but it's not even remotely close to what you're rallying for.

                      1. Stoneshop Silver badge

                        Re: Standards Bodies need notice

                        I don't agree with you on concept of Internet of trust where only allowed devices can access it as implications of that are too far fetched.

                        Trust, to the extent that "this device is configured with reasonable protection against remote attacks, which includes [list of security 101 measures]". This needs to be done to mitigate proliferation of Mirai c.s., and is by no means the one single solution required.

                        TLDR There is solution but it's not even remotely close to what you're rallying for.

                        I haven't seen any details of YOUR plan yet. Care to provide some, instead of muttering defaitist boilerplate?

                      2. Doctor Syntax Silver badge

                        Re: Standards Bodies need notice

                        "There is solution but it's not even remotely close to what you're rallying for."

                        I haven't seen you suggest it.

                        1. Stoneshop Silver badge
                          FAIL

                          Re: Standards Bodies need notice

                          I haven't seen you suggest it.

                          Oh, I found this:

                          The only solution for this particular issue is a protocol that can stop traffic towards victim at originating ISP level. Not that hard to do really.

                          Yeah, that totally doesn't require just about every* ISP on the planet to sign up for that, agreeing to some extension of a couple of very basic network protocols, upgrading their software and maybe even their equipment to accommodate that protocol, and figuring out a way to reliably determine which of those millions of network packets are actually malicious.

                          And never mind that, next to China being a major source of idIoT junk, there's also a lot of networking and telco gear manufactured there.

                          * If you can't get South American and Asian providers on board you'll have the same problem as with those countries not banning (and enforcing that ban) IoT stuff that essentially hollers "Pwn me!"

          2. JLV Silver badge
            Boffin

            Re: Standards Bodies need notice

            >strict requirements in EU and USA

            Market loss. Take these 2 out and your trinkets become a lot less profitable. C.f. Cyanogen becoming non-viable due to an India lockout.

            Fixing 80% of this problem is probably 20% of effort. Later they can worry about subtler things than factory default passwords

          3. Dan 55 Silver badge

            Re: Standards Bodies need notice

            If manufacturers are forced to follow standards in the west, they may as well do so for the rest of the markets. It gets more expensive to maintain two forks.

            Once tat is updated and tat from uncooperative manufacturers recalled, the west can legitimately begin to bring political pressure on foreign governments and economic pressure on foreign backbones. They might mirror the same certification steps in their countries or trading blocs to avoid this.

    2. DainB Bronze badge

      Re: Standards Bodies need notice

      That's just silly. How would you test for not yet known httpd or OpenSSL vulnerability ?

      You can hold anyone to any standard you want but you can't make a company that sold million routers with exploitable vulnerability and went out of business year later to fix anything.

      1. Voland's right hand Silver badge

        Re: Standards Bodies need notice

        That's just silly. How would you test for not yet known httpd or OpenSSL vulnerability ?

        There is a precedent - you cannot sell a car unless you guarantee that you will accept it for recycling and unless you provide spare parts for X years. While the laws which combine to form these reqs are different in Eu and US the net effect is the same.

        In any case, most of the insecure crap is resold with "brand labels" like Belkin, Dlink, etc and those are not going anywhere. In fact, let's hope that this incident contributes towards the reduction of "outsourcing your incompetence and putting a brand label on it".

        1. DainB Bronze badge

          Re: Standards Bodies need notice

          I just wonder if you notice subtle difference between $30K car and $50 electronic device and how differently both industries regulated.

          The only solution for this particular issue is a protocol that can stop traffic towards victim at originating ISP level. Not that hard to do really.

          1. Uffish

            Re: Standards Bodies need notice

            I put something like €50 of diesel into my car when I fill it up. The diesel fuel meets several standards that protect me but also specifically protect the population at large (sulphur content etc). The sale of non-standard fuel is illegal.

            Either you enforce some sort of legislation that makes IoTs less vulnerable or you live with an internet with the quality of service of Southern Rail.

          2. Doctor Syntax Silver badge

            Re: Standards Bodies need notice

            "I just wonder if you notice subtle difference between $30K car and $50 electronic device and how differently both industries regulated."

            Your $50 electronic device should already be regulated as regards electrical safety.

      2. Stoneshop Silver badge

        Re: Standards Bodies need notice

        You can hold anyone to any standard you want but you can't make a company that sold million routers with exploitable vulnerability and went out of business year later to fix anything.

        However, once the regulating bodies declare non-conforming* devices to be illegal and requiring them to be taken offline, the next step should be to legitimise ISPs using the Mirai code (and other means) to identify vulnerable devices. If end users don't respond to notifications that they're using uncertified crap, they need to be sandboxed or taken offline entirely.

        Drastic, yes, and needs law and regulation changes, as well as secure processes for upgrading certified devices, so it won't happen tomorrow, but to me it looks to be the only way to get rid of IoT shit that's vulnerable and can't/won't be upgraded.

        * certification includes having a way to patch in case new vulnerabilities are found.

        1. DainB Bronze badge

          Re: Standards Bodies need notice

          I'm not even sure if you're trolling or just unable to understand what you're proposing. Do you really want to live in communist utopia where government can control which device you can use to connect to Internet ?

          1. Stoneshop Silver badge

            Re: Standards Bodies need notice

            Do you really want to live in communist utopia where government can control which device you can use to connect to Internet ?

            Proposals to certify idIoT devices are nothing new, and equivalent regulations concerning wireless comms have been around for eight decades. This is to try to reduce the number of devices that are actively disturbing a particular communication medium, so not at all unlike the FCC and other agencies clamping down on inappropriate radio airwave use with bans and fines for using devices that lack certification.

          2. Doctor Syntax Silver badge

            Re: Standards Bodies need notice

            "Do you really want to live in communist utopia where government can control which device you can use to connect to Internet ?"

            I didn't see that being suggested. It's not a matter of controlling which device, it's a matter of controlling the safety standards they meet. They'll already by subject to all sorts of safety requirements. For instance the telecoms network operators will already have specs as to what can be connected to ensure it doesn't put harmful voltages on the line or draw excess current. Or are your telecoms providers communist-run?

  3. benderama

    I don't think you can say "we weathered the storm" AND "our systems are coming back online" in the same speech. If you're knocked offline, you did not "weather" an attack.

    1. Adrian Midgley 1

      Heaving to is the analogy there.

      Not making progress, but not sinking or breaking, is weathering the storm.

      Once the storm abates progress resumes.

      #philology

      1. yoganmahew

        Re: Heaving to is the analogy there.

        Well, maybe. Beached on a sandbank and refloated, if you want to use nautical terminology.

        1. DropBear Silver badge

          Re: Heaving to is the analogy there.

          No. All your convoys were sunk last month by U-boots, if you want to use nautical terminology. This month they seem to arrive mostly. The ones from last month are resting on the sea floor and didn't "weather" jack squat; they just went down.

  4. Arbeebee

    Home Router Traffic

    Although I doubt any of my devices decided to join in the fun, how would I know? Any particular ports on the router to monitor?

    1. Destroy All Monsters Silver badge
      Holmes

      Re: Home Router Traffic

      The graph of I/O bit-per-seconds?

      Unless it was manipulated too...

      I'm looking forward to an Advice Dog Meme flood on the subjet of IoT shit.

    2. CommodorePet

      Re: Home Router Traffic

      Mostly port 23 / Telnet. MIRAI looks for that port, tries a bunch of known hardcoded values, then it usually finds a busybox shell running on ARM. It can then run busybox commands to download additional scripts and apps that perform the DDOS at whatever target they desire.

      1. frank ly Silver badge

        Re: Home Router Traffic

        The ShieldsUP! tester at grc.com tells me that I have perfect stealthing, for the common ports at least. Does anyone know of any flaws in this form of testing?

        If I tether my laptop to my mobile phone, to use an internet access path from outside my domestic ISP, is there a 'probing' application I can use to check my home IP address for leaks and vulnerabilties? (I promise I'll only use it on my own home IP address.)

        1. Peter Gathercole Silver badge

          Re: Home Router Traffic

          The problem with Shields Up! is that by default it only checks the reserved ports 0-1023.

          You can use it to do custom scans, but the standard check will not check to see whether uPNP has opened up ephemeral ports through your firewall, and once these are set up, it could allow CnC channels to any devices.

          But most edge-firewalls allow outbound connections to a co-ordination server anyway (it really would be a pain to have to configure individual ports on the firewall), and once a session is established, will allow return control requests (remember TCP/IP sessions are bidirectional) even without uPNP (never wondered how your network attached, print-from-anywhere printer works? Well, this is it).

          Of course, it is necessary to get a foothold in the network for uPNP or outbound requests to be made, but who knows what is baked into the firmware of these IoT devices from China? I tent to run a Linux firewall, and do a sweep of the ports currently in use at the firewall, but it's difficult.

          It's all a bit of a mess. I favour using the vulnerabilities themselves to run destructive code on the IoT devices to break them, but that is illegal in pretty much all jurisdictions.

        2. Anonymous Coward
          Anonymous Coward

          Re: Home Router Traffic

          The ShieldsUP! tester at grc.com tells me that I have perfect stealthing, for the common ports at least. Does anyone know of any flaws in this form of testing?

          I am guessing it is potentially possible that a device could keep its ports closed until it sees a certain number of attempts to gain access, or access to a particular sequence of ports, or could advertise to the command centre its ready to accept command probes at specific times.

          It might be fairly easy to create a system which could fool GRC and other probes into thinking things are okay when they are not.

          Virgin Media have been telling - some would say scamming - customers into believing they are open to SSDP vulnerabilities even though they won't explain how that is and GRC shows no holes, and they say GRC showing no holes proves one is safe!

    3. Mage Silver badge

      Re: Home Router Traffic

      The most evil feature added after Autorun (Win95a and earlier Amiga) was uPNP, especially on routers!

      It should be illegal to have uPNP on a router/firewall and have internet without a firewall. It's only a partial mitigation, but would stop most of current IoT compromise.

      There is no complete solution.

      1. Metrognome

        Re: Home Router Traffic

        Sorry, wrong. uPNP and zero conf are godsends in home environments. Most peeps would rather have things work out of the box than worry if their fridge is actively DDoS-ing some far flung and mostly unknown entity that does "something on the internet".

        I remember the bad old days when I had to physically call up to my parents' house to re-work whatever mess they found themselves in. Now, uPNP, ports open automatically, things communicate and join each other and to be honest, they have precious little to secure on their devices much like most people.

        Also, to all the standards-talkers, persuade China first, discuss afterwards.

        Sorry to rain on your parade but some real-world perspective would help a lot of people understand the true limitations of the options they offer.

        1. Doctor Syntax Silver badge

          Re: Home Router Traffic

          "Also, to all the standards-talkers, persuade China first, discuss afterwards."

          No, require stuff legally on sale and/or in use to meet standards and China will be persuaded.

          1. Charles 9 Silver badge

            Re: Home Router Traffic

            China ships directly to you, AROUND standards. How do you stop that without a bureaucratic nightmare?

        2. Peter Gathercole Silver badge

          Re: Home Router Traffic @Metrognome

          UPnP.

          Convenient, yes.

          Secure, hell no.

          One thing it allows is any internal device to knock inbound holes in your firewall, without your knowledge or approval.

          I appreciate that without it, some consumers would have to learn something, but the downside is that all the IoT devices that sit inside home networks and use UPnP can potentially become a participants in a DDoS attack like this.

          Do consumers worry about this? Well probably none of them understand what it is that caused the DYN DNS outage, and even less about whether their house was part of the cause.

          But should we? Definitely yes, if we want to maintain a functional and usable Internet!

          I run my firewall with UPnP disabled, so it works inside my network for device discovery, but the firewall can't be controlled, and there's not that much that either I or the other members of my family have noticed that doesn't work.

          1. Metrognome

            Re: Home Router Traffic @Metrognome

            I'm fully with you on that. I operate under the assumption that everything is indeed compromised and then work backwards to see if anything was worth securing or not. But in the case of elderly relatives and associated clueless people, uPNP just works and it's good enough for them and their needs.

            As for DDoS'ing some unknown place, they couldn't care less and there's no way they'd start fiddling with any settings outside the default ones for the sake of no one. Hell, they hardly even do this for their own safety and on their own mobile phones, what makes you think they'd start now caring about DDoS-ing?

  5. stg

    Anyone surprised?

    http://www.passwordbingo.com/the-password-blog/2016/10/11/how-long-before-we-are-being-fined-for-having-lame-passwords

  6. John Crisp

    Campaign required

    Maybe El Reg can start a campaign a la 'Daily Fail' and then claim the credit if this monumental trip to Cockup City ever gets fixed :-)

  7. Number6

    What would help is for ISPs to cooperate and shut down customers who are clearly participating in the attack. Route 'harmless' http requests to a default web page explaining why they've been taken offline and what to do about it.

    I know such a feature could be abused, but I'm sure there would be a way round that with checks and balances and a proper procedure (yeah, right...)

    1. Anonymous Coward
      Anonymous Coward

      Oh absolutely. I've often thought about similarish schemes where you essentially stop a user in their tracks to force them to upskill (I'm fantasising, clearly), or address something.

      The problem as ever will be no company having the balls to do this. Fearing the paying customer backlash.

      I'd hate to be the phone jockey on THAT tech support call.

      The customer is not always right. I think this proves that with ownership comes a wider responsibility. It's like guns. You have to show you're responsible as the chance of misuse is lethal.

      Here, the chance of misuse is perhaps high, and boy, the botmaster behind this will be smug tonight w*nking himself silly over the sheer spread of problems the DDoS caused.

      1. Doctor Syntax Silver badge

        "The problem as ever will be no company having the balls to do this."

        Turn that one round. As one of Nixon's henchmen said, when you have them by the balls their hearts and minds will follow.

        Require them to do this.

        1. Charles 9 Silver badge

          You Can't REQUIRE a sovereign nation like China to do anything without a treaty. That's part of the definition of sovereignty.

    2. anoco

      I don't know about shutting down customers, but the ISPs could block their traffic to the affected IP very easily I think. Since the number of major ISPs is relatively small it would only take a few of them to make the attack manageable on the receiving end.

      A "DNS Alert" (similar to US Amber Alert) type of a thing alerting the 100 biggest ISPs on the planet would defang the attackers in minutes. You could even add another D to the attack description. DefangedDDoS

      OR.... deflect all the traffic to North Korea and watch their 3 servers go up in smoke, just saying...

      1. Charles 9 Silver badge

        Except there would be collateral damage. Those targets also have LEGITIMATE business via the web. You'd be doing the DDoS's job for them using that, and the way the IoT botnet works, they use the same legitimate requests we do, so they're camouflaged as well. As for the ISPs, they don't see a lot of traffic individually, and the amount they emit wouldn't probably surpass traffic from a home server running, say, a home camera feed.

    3. Ken Hagan Gold badge

      Would it be straightforward to limit domestic users to (say) one DNS query per second? Would this help?

      It's a well-known port, so the traffic would be easy to identify and handle separately. Domestic users are the most likely to be running dodgy IoT devices and the average domestic router ought to be configured with its own DNS cache anyway, so the throttling might not even be detectable by Joe User. It obviously wouldn't help against DDos attacks aimed at other services, but DNS seems to be a popular target, perhaps because the consequences are so spectacularly widespread.

      1. Updraft102 Silver badge

        "Would it be straightforward to limit domestic users to (say) one DNS query per second? Would this help?"

        DNS resolution is needed for a lot more than just the URL you typed into the browser or clicked in Google. Each of the secondary domains that site calls have to be resolved too, and there can be dozens of them on a fairly typical site on the internet. Running uBlock Edge, I see a counter that shows the number of third-party domains a site has attempted to contact, and it has sometimes exceeded a hundred of them. It's absolutely nuts (and much of it is about tracking and analytics related to advertising), but that's the state of things now.

        Not only that, but it would only work when it is a DNS server being attacked. That's not always the case.

        1. Doctor Syntax Silver badge

          "DNS resolution is needed for a lot more than just the URL you typed into the browser or clicked in Google. Each of the secondary domains that site calls have to be resolved too, and there can be dozens of them on a fairly typical site on the internet."

          To say nothing of the tertiary and quaternary domains. OTOH if this forced sites to serve all their own crap this could be seen as a useful by-product

  8. maccy
    FAIL

    "The blame is not with dyn"

    No, but it isn't with cheapo IoT either. DDoS seems to be built into the structure of the internet. Any system that relies on millions of components being "nice" is a system that is doomed to fail.

    1. Warm Braw Silver badge

      Doomed to fail

      Unfortunately, that's pretty much right.The Internet was designed assuming that the network was under central control and that the only threat came from the physical destruction of its infrastructure: the goal was simply for packets to be able to get through if there was a path of some sort or another to the destination.

      Today's Internet has no central control and the main threat comes not from the physical destruction of its infrastructure but the unwanted behaviour of systems attached to it.

      The "other" protocol that was briefly being touted for public networks (X.25) had provision ("the D bit") for end-to-end flow control at the network layer which is one of the things that could be used to mitigate problems of this kind. However, even where X.25 was widely deployed, I'm not aware of networks (reliably) using the D bit feature. That's largely because network flow control is a hard problem to solve and the queue-or-discard mechanism of the Internet works at least as well as anything else under "normal" circumstances.

      However, it's a problem that needs solving. There has to be a back-pressure mechanism that sends a "stop" to the ingress point since there is no practical means of ensuring that every piece of equipment in private hands is well behaved. That of itself is not a panacea - and is potentially a new route to DDoS by spoofing the back pressure - and, if you look at the IPv6 gestation period, unlikely to be with us any time soon. It's also not the only issue that needs attention - more privacy, anyone?

      1. Charles 9 Silver badge

        Re: Doomed to fail

        "However, it's a problem that needs solving. There has to be a back-pressure mechanism that sends a "stop" to the ingress point since there is no practical means of ensuring that every piece of equipment in private hands is well behaved. That of itself is not a panacea - and is potentially a new route to DDoS by spoofing the back pressure - and, if you look at the IPv6 gestation period, unlikely to be with us any time soon. It's also not the only issue that needs attention - more privacy, anyone?"

        Intractable problem. The ONLY reliable way to manage a network is to introduce ironclad attestation. But that instantly eliminates privacy. What's happening is that the wired world is reaching the "wishbone" point: a point in which the third option is disappearing from the strain exerted from both extremes (in this case, the Anarchy of the current Internet and the Police State of a Stateful Internet). The pressures mean ANY third option quickly slides into one or the other extreme, rapidly NOT becoming a third option. Eventually, the wishbone will break, meaning no third option is possible anymore because it'll IMMEDIATELY gravitate towards one or the other extreme (the "winner"). In which case, only three options will be left: Anarchy, Police State, or Walk Out?

  9. Anonymous Coward
    Anonymous Coward

    Blame people

    Blame people for not changing the passwords

    Or blame the iot manufacturers for hard coding passwords, which happens too frequently

    Blame the DNS services for not being prepared

    But most of all blame the Ooh Nice Shiny culture that allows the IoT to be big enough to be dangerous!

  10. CommodorePet

    Time to block port 23 (Telnet) for ever

    All ISPs need to disallow port 23 to and from consumers / end users. Nothing needs this anymore.

    1. anoco

      Re: Time to block port 23 (Telnet) for ever

      Why stop with 23 only? Get rid of most of them. 640 ought to be enough for anybody.

  11. dbtx Bronze badge

    Capt. Hindsight

    "All those routers and smart devices should have had randomized passwords printed on a sticker on the device-- not the same default password on everything. And companies that churn out smart devices should be hiring more white hats or grey hats to prove that they aren't trivial to compromise, and they should be spending more time and energy on making sure that they aren't creating a playground for criminals or providing for a snarky new sense of the word 'pre-owned.'"

    1. Dan 55 Silver badge

      Re: Capt. Hindsight

      Or ask for a password on first login from the local network, won't accept a crappy password (in the dictionary, in a common password list, or too small), and won't connect to the Internet until it gets a password.

      The username should not be something stupid like root or admin because that's half the job done.

      And most things shouldn't use a bare username/password combo if they're going to be accessed by apps or the mothership which make things automagical for the user, ssh/https + certificate + password should be used instead.

      1. DainB Bronze badge

        Re: Capt. Hindsight

        As long as you are happy to pay manufacturer to have support team that will be resetting these passwords 24/7. Are you ?

        1. Dan 55 Silver badge
          Facepalm

          Re: Capt. Hindsight

          If it's automagical app/mothership/cloud thing, the manufacturer has the password anyway.

          If it isn't, heard of a reset button?

          1. Charles 9 Silver badge

            Re: Capt. Hindsight

            "If it isn't, heard of a reset button?"

            Uh...no.

            What now?

        2. Doctor Syntax Silver badge

          Re: Capt. Hindsight

          "As long as you are happy to pay manufacturer to have support team that will be resetting these passwords 24/7. Are you ?"

          The user sets those. The default password is on the label. You reset it to get that and you then have to set a new password before you can get it online.

          You, the user, lost the label? Sorry, can't help you, we don't have a record of it.* You'll have to buy a new one. Please look after that better.

          * That prevents anyone ringing up trying to get the default password if it transpires the pile of crap device can be reset remotely.

  12. oiseau Silver badge
    Facepalm

    Today the web was broken ...

    Well ...

    T'was only a matter of 'when'.

    And 'when' came today.

    Believe me, it's only going to get worse.

    1. ecofeco Silver badge

      Re: Today the web was broken ...

      *sigh* Yep.

    2. Doctor Syntax Silver badge

      Re: Today the web was broken ...

      "Believe me, it's only going to get worse"...

      ...before it gets better.

  13. Mark 85 Silver badge

    Round 3 has hit....Lo

    Looks Dyn is under attack again tonight and the outages are spreading.

  14. Kevin McMurtrie Silver badge

    The blacklist of things

    Too many ISPs and networks can't be bothered to handle abuse complaints, especially if they think they're too important to suffer any consequences. If we're going to put billions of Idiot of Things into the IPv6 address space soon, this all needs to be automated. That means a standardized and automated means of reporting problems, automated means for an ISP to detect that the problem is happening, and the ability to automatically block customers until the problem is resolved. Networks who can't be bothered to participate or who create false-positives should be entered into public voluntary routing blacklists, much like those used to track spam-friendly ISPs. It's the only way to handle tens of millions of attacks quickly. (Google may go offline for a week when this is implemented, but I'm sure they'll get around to joining the party.)

    1. Charles 9 Silver badge

      Re: The blacklist of things

      Then what happens when innocent users SUE for the collateral damage of them not being able to go on the Internet for no fault of their own? And no, many of them can't switch ISPs, or those ISPs are blocked, too.

      1. Stoneshop Silver badge
        Mushroom

        Re: The blacklist of things

        for no fault of their own?

        They bought and deployed a cruddy, vulnerable IoT device, though that's just part of the problem. Those devices not bearing a label "CONNECTING THIS SHIT WILL FUCK UP YOUR INTERNET" is also a factor.

        1. Anonymous Coward
          Anonymous Coward

          Re: The blacklist of things

          And they don't care. That's not what they bought it for.

      2. Doctor Syntax Silver badge

        Re: The blacklist of things

        "Then what happens when innocent users SUE for the collateral damage of them not being able to go on the Internet for no fault of their own?"

        What happens? The ISPs learn the advantage of making sure it doesn't happen again. Or, to put it another way, they learn the cost of not having made sure it couldn't happen in the first place.

        As per another of your posts, we;re dealing with Stupid here so we need to to take actions that don't depend on Stupid understanding things.

  15. Old Handle
    Facepalm

    This probably show my own ignorance more than anything, but why is it that something like this has such severe effects? Doesn't DNS get cached in various places? It seems like a relatively short outage like this could be smoothed over almost completely taking advantage of that. At least for large sites which ISPs have doubtless accessed many times over the past hours.

    1. ecofeco Silver badge

      It's rather complicated and yet simple at the same. I know that doesn't make sense but I'm mobile right now and can't type out the lengthy explanation.

    2. Jordan Davenport

      One of the biggest problems is that people tend to use DNS for load-balancing across servers and set really short TTLs, so the cache expires in minutes if not seconds.

    3. DainB Bronze badge

      Maybe because Dyn stands for Dynamic and to be able properly do that they need really short TTL ?

      1. Jordan Davenport

        I guess some people can't follow a conversation thread unless you quote the person above. I was answering the original poster's question about why the caches didn't hold over until the service outage was over.

    4. SImon Hobson Silver badge

      > Doesn't DNS get cached in various places?

      Yes, but ...

      This malware probably doesn't use the local cache - in fact it specifically won't because that would mostly defeat the purpose of the malware. It most likely looks up (or is told) the IP address(es) to attack and sends it's DNS queries directly to there - thus bypassing DNS caches completely.

      Of course, ISPs (and savvy ned users) could block outbound DNS except to/from certain addresses - from the ISP side, they could restrict users from using anything but the ISP provided resolvers (and then get a right slagging off from tech savvy users fed up with the sort of crap resolvers many ISPs run). Even then, all the bot needs to do is generate multiple requests (eg a.target.tld, b.target.tld, ...zzzz.target.tld, aaaaa.target.tld, ...) and the cache is significantly less effective as each new request will cause a new lookup to the targets DNS servers - though this will also affect the resolver due to the rapidly increasing size of the cache.

      But that's only DNS, you can't block (for example) HTTP traffic without effectively killing the internet !

      This is where the botnets score. If a single device (or small number of them) is sending massive amounts of traffic then that's relatively easy to spot and block. But with a distributed attack (the first D in DDoS), you only have to generate a small amount of traffic from a huge number of devices. If done right, the DoS traffic is undetectable in amongst the legitimate traffic from the end users hosting the infected device.

  16. Andrew Jones 2

    Maybe as a starting point going forward:

    Devices ship with NO username and password. As part of the Setup process they ask you for a username and password which is then from that moment on the username and password for logging in to the device. It's not a huge issue if it gets lost or forgotten because just like now - a simple device reset starts you back at out-of-box.

    Maybe also devices should be "connect to the cloud" with opt-in as part of the setup flow, instead of by default.

    Finally - say this is all the fault of IoT because it's already been mentioned that both Routers and DVRs which are not typically badged as IoT devices are part of this as well - it's not as simple as people bought useless items and installed them. As we saw the other day AVTech which appeared to follow all the steps to make it not insecure by default - did really really stupid stuff like assume that no-one would ever write certain keywords into the URL so it was perfectly acceptable to execute commands received directly from the GET address - and execute them as ROOT no less. Clearly not just stupid but batshit insane, but the point is - I as a user did my job, I changed the default username and password and I did not port forward internet connections to it (I use a VPN instead). With all my Panasonic network cameras that we've had since 2006 - not one was ever allowed to access the panasonic dynamic DNS thingy, and none of them allow incoming connections from the net.

    But we know there is kit out there that does "helpful" things like negotiate UPnP to setup port forwarding and doesn't ask for permission to do so - and these are just some of the devices that need closely watching.

    Finally - as to the suggestion of arresting USERS because they have insecure IoT kit - that's stupid, there is no way that could ever be proposed to be added to law - the only arrests should be toward the company that created said insecure IoT kit, but you can never expect the end user to know if it is or isn't secure.

    1. DainB Bronze badge

      Good idea. Now get it implemented worldwide. Yes, I meant those basement manufacturers in China as well.

    2. Doctor Syntax Silver badge

      "Finally - as to the suggestion of arresting USERS because they have insecure IoT kit - that's stupid, there is no way that could ever be proposed to be added to law"

      That depends on how bad the problem becomes. There are several points to apply pressure.

      One is the market place via the types of regulation and certification that's in place already for electrical safety etc. It gives Trading Standards or the like to deal with vendors in the country and for customs to turn away incoming shipments. There's absolutely nothing novel in principle about this, it's just that govts. need to be kicked into motion to get a round tuit.

      Another is the ISPs and through them the users. They can be required to put it into T&Cs that non-compliant kit can't be exposed on the net, either outside of firewalls or via uPnP.

      Finally, after due warning, the users themselves if they insist on connecting stuff it can be made an offence. In practice, of course, the ISP would almost certainly deal with it by cutting off the customer but having the illegality as back-up to deal with awkward customers.

      All this combined would make non-compliant stuff unsaleable. That would lean on the manufacturers more effectively than trying to negotiate international standards.

      That leaves countries that are reluctant to get round to doing such things. "Nice internet connection you have there. Shame if it got disconnected for an hour or two now and again. Or a day or two."

  17. croc

    And we've been banging on about SCADA devices for how long?

    Shirley, SCADA devices are even more critical to lock down. Have we done so? Well, pretty sure that the Iranians have, but the rest?

    Bang on all you want.

    1. Anonymous Coward
      Anonymous Coward

      Re: And we've been banging on about SCADA devices for how long?

      I'm not sure that the Iranians have. Their engineering prowess is influenced by "honor society" models of thought, so they will fix the symptoms, not the causes.

  18. ecofeco Silver badge

    Top notch reporting

    Excellent breakdown and summary.

    Good job El Reg!

  19. Anonymous Coward
    Anonymous Coward

    What moron puts all their DNS servers on one network (Dyn, in this case)? This has been a known bad practice for decades, and the consequences are as old as the first time microsoft.com vanished from the Internet in the 1990s for the same reason.

    This is as much an architectural failure and management blunder as it is a diabolical attack.

    1. Nate Amsden Silver badge

      Your foolish comment makes it sound like dyn is not a globally distributed network of servers.

      It's not the same as pointing to a pair of fucking bind boxes on 2 subnets.

      It's 1000x more resilient than that.

      I'll bet that this is the largest attack in history. If they can take down dyn then they can take down just about anyone.

      Even amazon uses both dyn and ultradns and they had issues too.

      (Enterprise dyn customer for 7 years and been running my own dns for 20 years)

      1. Nate Amsden Silver badge

        This time it was just one provider though there were massive effects on isps that dyn uses as well.

        Next time they may target root dns servers. Take a few of those down and you really fuck the "phonebook of the internet"

  20. Anonymous Coward
    Anonymous Coward

    Dyn = Single Point of Failure

  21. Ken Moorhouse Silver badge
    Coat

    I'll get the popcorn-maker out

    Internet-connected, of course

  22. GrapeBunch Bronze badge

    Parallel Universe of IoT hurt

    as smart fridges and their IoT ilk get slotted in via "Smart Meter" routers, over which punters may have no control. Incidentally, can any North American "Smart Meter" user report whether these things sport a CSA, UL or FCC logo?

    1. Anonymous Coward
      Anonymous Coward

      Re: Parallel Universe of IoT hurt

      "smart meter" is an oxymoron, whilst it may provide info in a digital format it's just another gov't backdoor into your life. Plod already get informed about suspicious energy usage, hydroponic growth lights are a classic, imagine their whorgasums when they can tell you've gone to the fridge to get the butter for some late night adult fun...

      1. Pedigree-Pete
        Joke

        Re: Parallel Universe of IoT hurt

        Late night tea and crumpet? Disgusting behaviour.

  23. Ken Moorhouse Silver badge

    A few points

    1. Having an automatically generated Username/Password is what many manufacturers do right now (routers and access points, for example). However, there have been cases in the past where the algorithm/mapping that generates such combinations has been cracked.

    2. Getting ISP's to block dubious traffic: this has been the case with SMTP and file-sharing traffic ports for some time now, but there are many ISP's that don't, and there are good reasons why they should not get embroiled in such matters.

    3. CE-marking and US equivalents are good for purchasers to aspire to buying, but faced with a choice between high-price or low-price, with the difference in features being purely a few regulatory stickers affixed to the casing, which one will the purchaser end up buying?

    4. I think, and I've said this before, the Anti-malware companies need to target this as a compelling USP, in collaborating with the body that assigns MAC addresses: Build a database of devices, maintain a directory of firmware signatures associated with those devices and use that within the perimeter of LAN's to effectively poison those errant devices. I appreciate MAC addresses can be changed, but as a proportion of total devices, is it going to affect a DDOS attack? Those with the ability to do such things will also have the ability to ensure their IoT device is well-maintained.

    1. Anonymous Blowhard

      Re: A few points

      "CE-marking and US equivalents are good for purchasers to aspire to buying, but faced with a choice between high-price or low-price, with the difference in features being purely a few regulatory stickers affixed to the casing, which one will the purchaser end up buying?"

      The point about having legally enforced certification requirements is that you don't give the punters a choice about buying certified or non-certified equipment; you use the law to prevent the sale of non-conforming equipment. There will always be a small grey market, and an even smaller black market, in non-conforming equipment but 95% of buyers will get their gear from ISPs and responsible retailers.

      Yes it is an international problem, but I don't think the major Internet nations (Europe, US, China Russia) are unable to enforce this on their home turf, and it is in everyone's interests to have a smoothly functioning Internet. It can't be done overnight, and the alternatives are that governments make ISPs put limits on what you can and can't do with your internet connection (e.g. preventing you using your connection to host servers by blocking ports to discourage botnets) and no-one wants that.

      1. Anonymous Coward
        Anonymous Coward

        Re: A few points

        "Yes it is an international problem, but I don't think the major Internet nations (Europe, US, China Russia) are unable to enforce this on their home turf, and it is in everyone's interests to have a smoothly functioning Internet."

        You ever thought counties like Russia and China DON'T want a smoothly functioning Internet? At least, not one that smoothly functions outside their own borders, in which case who cares what happens? The less Internet out there, the less subversive material that can possible get smuggled in. In other words, they'd have an active interest in NOT policing this stuff...because Western companies collapsing is tacitly to their benefit.

    2. dbtx Bronze badge

      Re: A few points

      "However, there have been cases in the past where the algorithm/mapping that generates such combinations has been cracked."

      If the device is giving up any information before authentication which can be used to uniquely identify a device (besides its MAC) and in turn find out what the auto-generated password should be, then clearly someone is doing something wrong and needs to stop.

      If the industrial thing that picks a MAC from the unused pool is the same thing that generates a password, and it does both at the same time in some deterministic & connected way, then clearly someone is doing something wrong and needs to stop.

      Other than that, what "mapping" is there? AFAIK the only thing anyone should be able to figure out from the login page is the model number.

      1. Ken Moorhouse Silver badge

        Re: A few points

        Here's a typical example of what I meant:-

        http://www.gnucitizen.org/blog/default-key-algorithm-in-thomson-and-bt-home-hub-routers/

        1. dbtx Bronze badge

          Re: typical example

          I don't want to live on this planet anymore.

    3. Doctor Syntax Silver badge

      Re: A few points

      "CE-marking and US equivalents are good for purchasers to aspire to buying, but faced with a choice between high-price or low-price, with the difference in features being purely a few regulatory stickers affixed to the casing, which one will the purchaser end up buying?"

      The alternative should be between the device being legally offered for sale or not. That doesn't provide the buyer with much of a quandary. If he buys from Del-boy he risks the device being forfeit, and maybe a fine.

      1. Charles 9 Silver badge

        Re: A few points

        But Joe Stupid can't tell the difference between one legally offered for sale and one ILlegally offered. They don't know enough to beware and never will. Remember car boot laptop sales?

  24. Anonymous Coward
    Anonymous Coward

    Null firmware

    Would it be possible, if you know the access/admin to all these devices, and know they're being exploited when they start taking part in a DDOS, to write a script that determines the IoT toe and defaults and then either:

    1) triggers a reboot

    2) flashes default firmware

    3) resets the passwords/admin account to random

    4) flashes Null firmware essentially rendering the device junk

    The last two would be preferable as they stop the device being used for attacks so easily, as a side effect forces the end users to confront the fact the IoT shit they are using has been compromised ("other people can control my camera? But it watches my child/house/sexy-sexy times with my goat") and in my view the major benefits would be the slew of support calls to the sellers of this badly secured cruft, shouty reviews on Amazon, etc.

    1. Anonymous Coward
      Anonymous Coward

      Re: Null firmware

      Support lines would be nonexistant and the company exists in China so no law enforcement can touch them. As for shouty Amazon reviews...countered by shill reviewers or threats.

  25. Anonymous Coward
    Go

    The only way to really fix this...

    ...is to get some graphic designers right now to draw-think a logo and some 'digital evangelist marketeers' to focus on a pithy name and strapline for whatever the problem actually is. And we need lattes. Chai lattes if we're really going to get on the good foot.

    Once we have our call to action defined, and the internet starts working again, we can then realise the actual campaign work: forging a dedicated social presence to make this problem join the conversation now!

    We have Macbooks. We have man bags. We can curate this together.

    (PS can anyone explain what a router is? Is it the bit of Siri that does the maps?)

    1. Archtech Silver badge

      Re: The only way to really fix this...

      Don't forget that vital necessity: A Manifesto!

  26. Mr Dogshit

    Good luck with persuading the Peoples' Golden Chrysanthemum Happy Panda Electrical Concern to make secure products.

    1. Uffish

      Re: persuading

      Easy - get a large slice of the market to outlaw the sale and use of insecure products and enforce the ban. Shortly afterwards there will be secure Golden Chrysanthemum gadgets on sale - at the appropriate price.

      Legislation that is enforced is the key - it's not Happy Panda's problem, it's ours.

      1. Anonymous Coward
        Anonymous Coward

        Re: persuading

        How do you enforce laws when sovereignty gets in the way? Especially hostile sovereignty that can simply ship things around customs?

      2. Doctor Syntax Silver badge

        Re: persuading

        "it's not Happy Panda's problem, it's ours."

        It's theirs if they can't sell their stuff. Contains full of instant land-fill being turned away at the docks? The message will get through PDQ.

        1. Charles 9 Silver badge

          Re: persuading

          Or they could retaliate with sanctions or suits at the trade court.

  27. Adrian Midgley 1

    Running your own DNS and using hosts files

    down to a local level seems more useful today than last week does it not?

    1. Anonymous Coward
      Anonymous Coward

      Re: Running your own DNS and using hosts files

      Not really. You need a way to refresh the entries, which is why DNS is hierarchical. But I suspect the attackers will find a way to bring the entire Internet to its knees. Even if they have to attack 20 places at once, with a million devices to each one, they could still do it. What next? Carpet-IP-bombing by setting every pwned devices to flood the Internet with random honest-looking HTTP requests that can't be distinguished from real ones?

  28. Anonymous Coward
    Anonymous Coward

    For the good of mankind please take down Facebook, Twitter was a good start.

    1. Adam 1 Silver badge

      Give Musk some credit to that end.

  29. Anonymous Coward
    Anonymous Coward

    Remove SPOF

    It's always annoyed me that most DNS providers do not allow zone transfers thus it is not possible to say sign up with a number of dns providers, have your master server with one (or indeed on your own server) and then configure each suppliers servers as slaves which only accept/request updates from the master.

    If this was allowed it would seem you could limit the effect of dns issues as hopefully at least some of your providers would be online.

    Having a single supplier for dns has always struck me as flawed regardless of the claims from those suppliers of how fantastic their infrastructure is.

    1. Anonymous Coward
      Anonymous Coward

      Re: Remove SPOF

      I consider "Zone Transfers" as a relic, frankly. What's wrong with pushing updates using a REST interface to each server that needs to publish the zone?

      1. Adam 52 Silver badge

        Re: Remove SPOF

        We're in the midst of a huge great DDoS attack, and you're seriously suggesting opening up DNS servers to port 80/443?

      2. Nate Amsden Silver badge

        Re: Remove SPOF

        Like the one dyn has?

        Good luck getting that standardized across dns implimentations and providers.

        Zone transfers and dns notify seems to have worked very well for me for the past 20 years I have no reason to use another method.

  30. Hans 1 Silver badge
    Mushroom

    >Unlike your PC or your phone, IoT devices don’t have the memory and processing to be secured properly, so they are easily compromised by adversaries and it’s very difficult to detect when that happens.

    No, no, no, no, no, no, no! Oh Mamma-mia, Mamma-mia, Mamma-mia let him go! That is the BS at its worst. Shut up, do some other job, you, sir, are not fit for working in IT. Christ!

    As if processing power is required to block telnet or do away with hard-coded passwords.

    All IoT devices MUST have open source software, must be update-able over the network, and perform the update from secure servers, look for updates on a weekly basis. All above and future problems solved. Don't adhere to this, don't get a license from FCC, EU etc

    1. Charles 9 Silver badge

      "All IoT devices MUST have open source software, must be update-able over the network, and perform the update from secure servers, look for updates on a weekly basis. All above and future problems solved. Don't adhere to this, don't get a license from FCC, EU etc"

      The devices come from China and are imported direct. Who gives a damn? As for the update mechanism, they'll just hijack it and pwn it THAT way.

      1. Destroy All Monsters Silver badge

        Do androids dream of botnet-zombified electric sheep?

      2. Doctor Syntax Silver badge

        "The devices come from China and are imported direct. Who gives a damn?"

        Market traders if they're importing them when Trading Standards come calling.

        ISPs when they're exposed to fines for routing non-compliant stuff. As I said in another post, there are multiple points to apply pressure to make stuff unsaleable.

        1. Charles 9 Silver badge

          And ways around them, too, like false flagging and knocking off.

  31. TeeCee Gold badge
    Meh

    "....big names including GitHub, Twitter, Reddit, Netflix, AirBnb ...."

    Was anything affected that I might actually give a flying fuck about it disappearing for a while?

    1. Mage Silver badge
      Coat

      Re: "....big names including GitHub, Twitter, Reddit, Netflix, AirBnb ...."

      No, but could be next time. They might come for El Reg.

      "First they came for ..."

      1. Adam 1 Silver badge

        Re: "....big names including GitHub, Twitter, Reddit, Netflix, AirBnb ...."

        > They might come for El Reg

        Distributed Denial of DevOps?

    2. Nate Amsden Silver badge

      Re: "....big names including GitHub, Twitter, Reddit, Netflix, AirBnb ...."

      For me it took out our company's website. Even the euro hosted version. The dyn servers in europe were hit hard too as my monitoring for europe website originates at only european locations.

      Add to that our external notification system pager duty was hit hard too and unable to function. Was getting a lot of calls from them (automated) with no content others said they just heard the message "applocation error"

      West coast US dyn was hit as i wqs quering west coast servers for pager duty dns but got no response.

      Users couldn't login to datacenter vpn because well dns was out. I happened to be physically on site (one of two trips per year). Our chat app is slack and maybe coincidence but could not login to slack on my computer from our data center for 30mins it just hung with no error message.

      Management is considering adding a 2nd dns provider I told them obviously if we do that than a 2nd CDN provider is needed and would be a good idea to have a 2nd external monitoring provider as well. All comes down to costs.

      Dyn's track record is practically flawless over the past 15 years(I've been an enterprise dyn customer for 7 of those). They know what they are doing. Myself anyway has to cut them some slack. They do have a 15 second SLA though ddos may be an exception to it.

      It's also obvious they will be doing tweaks to their strategy to help combat this better in the future, and obvious not to expect this can't happen again. Other than IoT botnets it's impressive to me dyn has lasted thisnlong without serious outages. Amazon became a dyn customer roughly 6 years ago after a massive ddos on ultra dns. (They still use both today though there are more dyn servers in their whois record than ultradns when I looked yesterday )

      Fortunately at the end of the day the attack was little more than an annoyance for me personally. I am more facinated by the scale of this attack than anything else.

      All in all dyn responded quite well. I have been involved in outages that have stretched more than 30 hours (and being awake on it the whole time). So I have battle scars and a few hrs of disruption doesn't make me blink anymore.

      My org was involved in collateral damage from another round of ddos that targeted internap earlier in the year. That took probably 2 weeks before it was completely dealt with? That too was by far the biggest ddos impact i had seen from internap (who has a 100%uptime sla though ddos not covered) in being a customer for 10 years.

  32. Archtech Silver badge

    Why??

    'A spokesperson for US Homeland Security said the agency is "investigating all potential causes" of the mega-outage'.

    Why are they going to all that trouble? Obviously, Putin did it.

    1. Kiwi Silver badge
      FAIL

      Re: Why??

      Why are they going to all that trouble? Obviously, Putin did it.

      Nah, was Shillary.. Just ask Chump, she has the nation's DNS running on her private illegal servers which she's bribed the FBI et al to ignore...

      Icon - the US for letting them (especially the hair-freak) stay around so long.

  33. Kaltern

    Skynet is coming.

    I'm genuinely worried about this, as this really was a easy, small and well coordinated attack on one company. Clearly there is no redundancy for Dyn, and companies reliant on it's services obviously didn't take any threat seriously enough to make arrangements.

    What would happen if the same group, did this again, but instead on attacking one company, they attacked a larger number, with an even bigger botnet. What would happen if they rotated the servers being attacked, and the botnet attacking them on a regular basis? How the hell would that be stopped, short of cutting ann DNS servers - even if that was a viable last-ditch defense?

    I can see this attack being nothing more than a test run. The 'big' attack is coming.

    1. Anonymous Coward
      Anonymous Coward

      "Clearly there is no redundancy for Dyn"

      Or Dyn has redundancy but the botnet was able to swamp the redundancy just as easily. Which would be even scarier as you're talking tsunami-type Internet flooding capable of swamping a provider who lives on mitigating swamping attacks. Like building for a once-in-a-century wave only to encounter a once-in-a-millennium wave.

  34. Sadie
    Paris Hilton

    Education

    So what steps can a typical home user take to secure their network. Say for example BT HomeHub or other ISP router with say Wifi/Cloud Printer, ISP provided TV Box, Internet Radio, Games Consoles, Phones,Tablets, Kindles etc. All branded, No devices from the Biou-Qing Mat-Toilet Company of Luquio Taizhou ;).

    1. IT Poser

      Re: Education

      That is the question that brings me back to ElReg. Unfortunately it appears there is no easy answer. Simply doing smart things like using script blockers and avoiding windows like the plague it won't fix IoT problems. It appears that, just like thermonuclear war, the only option is to not play the game. At least you know your vintage toaster isn't out to get you.

    2. Ken Hagan Gold badge

      Re: Education

      Fair question. Here's one answer. I'm sure knowledgeable people will chip in if I say something wrong ( https://meta.wikimedia.org/wiki/Cunningham%27s_Law ) .

      Make sure your router has its firewall enabled.

      Make sure that firewall is not allowing any incoming traffic.

      Make sure your router is not supporting UPnP.

      I would like to think that these are the default settings for any socially responsible router, but I fear that UPnP is probably enabled by default to enable attacks like we've just seen. (Oh, and also to enable world+wife to watch your webcam to see if you habitually pad about naken at home.)

      Your router definitely should have these features. If you can't find the controls for them, get a new router. If you can't get a new router, get a new ISP. If you can't get a new ISP, move house.

      1. Charles 9 Silver badge

        Re: Education

        "Your router definitely should have these features. If you can't find the controls for them, get a new router. If you can't get a new router, get a new ISP. If you can't get a new ISP, move house."

        And if you can't move house?

        1. Ken Hagan Gold badge

          Re: Education

          If you can't move house, set up a kickstarter or similar for an standalone firewall box that meets all the above requirements and which plugs into your ISP's LAN port. Then, once you've sold a few million to grateful end-users who have been frustrated by the quality of routers mandated by bottom-feeding ISPs, cash in the company and move house.

          1. Charles 9 Silver badge

            Re: Education

            "Then, once you've sold a few million to grateful end-users who have been frustrated by the quality of routers mandated by bottom-feeding ISPs, cash in the company and move house."

            And WHEN (not IF) your Kickstarter fails because all you're hearing are "squeaky wheels" and the average Internet-goer really doesn't give a soaring screw about what their stuff does, they just wanna go online, thank you, and many of them don't own or drive cars so won't get the driver's license analogy, either?

            1. Stoneshop Silver badge
              Holmes

              Re: Education

              And WHEN (not IF) your Kickstarter fails

              Then you build just one.

              1. Charles 9 Silver badge

                Re: Education

                You're talking people who wouldn't know how to program a VTR back in the day. Five words: Good Luck...You'll Need It.

        2. Stoneshop Silver badge
          Mushroom

          Re: Education

          And if you can't move house?

          "There is no problem that can't be solved by the judicious use of high explosives"

  35. Camilla Smythe Silver badge

    London Not Calling.

    I see the area around London is turning speckled red on the Level Three Outage Map.

  36. Andy Non
    Mushroom

    Maybe a sledgehammer approach is needed?

    If manufacturers are going to continue turning out IoT devices with little or no security to prevent them being used in these attacks, then ultimately they could be deemed a thread to the national security of many countries and should be treated as hostile. In which case nation state cyber warfare units may be justified in attacking and bricking all such vulnerable devices. Granted it would piss-off owners of such devices, but it may be the only way to get manufacturers to harden their security and prevent these devices from being conscripted. Ultimately, with enough insecure IoT devices they could be leveraged to bring down the entire Internet with devastating economic consequences.

    1. Anonymous Coward
      Anonymous Coward

      Re: Maybe a sledgehammer approach is needed?

      "If manufacturers are going to continue turning out IoT devices with little or no security to prevent them being used in these attacks, then ultimately they could be deemed a thread to the national security of many countries and should be treated as hostile."

      Problem is China can fight back, and China has nukes...

    2. IT Poser

      Re: Maybe a sledgehammer approach is needed?

      I know it isn't a real world solution but I'd love to see every infected device bricked. Since these device are already easy to access it should be fairly trivial to pop in and check for Mirai. Fry the device and let the owner try to get compensation from the manufacturer for selling a defective product.

      Obviously this isn't a long term solution as the Black Hats would quickly work out the attack vector. At least millions of compromised devices would be permanently removed as a threat.

      1. Charles 9 Silver badge

        Re: Maybe a sledgehammer approach is needed?

        There's also the matter of the manufacturer disappearing in the night, making it impossible to seek compensation.

        1. IT Poser

          Re: There's also the matter of the manufacturer disappearing in the night

          Buyer beware. If security experts hadn't been warning of this problem for years I could be convinced to care. As long as consumers don't face consequences from their bad purchase decisions they will continue to shop on price alone.

          1. ecofeco Silver badge

            Re: There's also the matter of the manufacturer disappearing in the night

            Your average consumer has no clue about things IT related. Nor will they ever and it's unrealistic to expect them to.

  37. Captain Badmouth
    Paris Hilton

    Chinese crap

    We're not even regulating the amount of crap that comes in now. What hope when govt. is cutting back on personnel in all these regulatory depts. and has done so for years. Gordon Brown started a big cutback in customs and excise, merging it with tax etc. We all know what a success that has been.

    I've lately had problems getting a govt. dept. to admit responsibility for overseeing imports of led shop signs wrt electrical safety. I even had to show them the relevant part of govt. legislation ffs!

    Good luck everybody.

    Paris : knows how to regulate her imports.

    1. DainB Bronze badge
      Joke

      Re: Chinese crap

      You'll need a lot of customs personnel to inspect all TCP packets crossing border of your country.

      1. Captain Badmouth
        Happy

        Re: Chinese crap

        When I were a lad TCP was something you put on cuts.

        Edit: It still is.

  38. Captain Badmouth
    Pint

    Late news

    Just on BBC news channel, the perpetrators have said that the outage was just a "test" and have threatened to "attack Russia if they messed with the US".

    I'll have a large popcorn with my pint, please.

    1. Captain Badmouth

      Re: Late news

      New world hacking is the responsible agency, apparently

      1. Destroy All Monsters Silver badge

        Re: Late news

        threatened to "attack Russia if they messed with the US"

        Excellent. A cleaner is probably on the way to Sheremetyevo International Airport.

        The miscreants' body will turn up swminning in some brownsite pond.

        End of story.

  39. gregthecanuck

    ISP - do they have the tools...

    Does any ISP have the capability to detect when their users are hosting a "floodbot"? To me this would be a good first step. If Joe/Jane homeowner gets hacked they get blocked until their offending device is taken down.

    Surely there must be tools for this?

    1. Charles 9 Silver badge

      Re: ISP - do they have the tools...

      Not really. Each individual contribution is not that big, so it's a form of "smurfing." It's only when taken as a whole that they're formidable. Like army ants and killer bees.

  40. Anonymous Coward
    Anonymous Coward

    no internet

    "And then make it illegal to run a vulnerable device if it's connected to the net."

    internet down, rings supplier, Hi, I've no internet connection can you see what the issues, Yes sir, you have a xyz webcam which has been deemed insecure and whilst it is connected we are required to disable your internet, But I don't own an xyz webcam, or indeed any webcam, gadget etc, just have my router which you supplied and my pc, I'm sorry sir our records show we have detected an xyz webcam on your connection and we are required to disable your internet until you remove it and send it to use for destruction, but I don't.....

    1. Charles 9 Silver badge

      Re: no internet

      "Well I'M sorry, but if do not remove this false claim immediately, I'll assume you're denying me service under illegal pretense, in which case you'll NEXT be hearing from my attorney."

      THAT'S why they don't do it now. They risk getting thrown in court if they're wrong.

      1. Nate Amsden Silver badge

        Re: no internet

        They don't do it now because it's too error prone and expensive for consumer connections.

        Datacenter network providers do this though. A couple of years ago for my personal server in hurricane electric they emailed me saying my server participated in an ntp amplification attack. I didn't believe them. But they very well could of been right as the out of band management port hasa NTP CLIENT on it. Little did I realize the implementation was poor in that it was also acting as a NTP server as well(and was vulnerable). Disabling the client closed the hole as there was no way to change the config(other than just turn it off ).

        They later contacted me again about another vulnerability but that time they were wrong as the ip that did the attack was not one assigned to me.

        I'd love to have a firewall in front of my server but i am allowed just 1 power outlet. I do run an inline openbsd firewall in a VM between my linux and windows VMs and the internet)

      2. Doctor Syntax Silver badge

        Re: no internet

        "in which case you'll NEXT be hearing from my attorney."

        In which case we'll produce stills from the camera as evidence.

        If you expose a camera on the web it's hard to deny that it's there.

        1. Charles 9 Silver badge

          Re: no internet

          But then what happens when the reply is, "But that's not my house. Ask the police." and shows them a picture of THEIR interior, which doesn't match?

  41. s. pam
    Terminator

    Welcome to the SkyNet party

    As IoT was always going to end up in a totally fscked 'Net.

    @InternetOfShit warned this was coming, but no one listened.

  42. WebLogons

    Inoculation?

    So how much trouble would someone get in, if they used the same attack vector, but just searched out suitable devices, open up the access and then change their password to something random.

    Job done surely, all potentially recruited devices now immune?

  43. Cashpot

    Too simple solution?

    There are apps on line now that link to thousands of web cameras using default settings to broadcast pictures of everything from sea views to children's bedrooms! The average punter is simply too stupid to be expected to change their password and let's be honest there is no incentive to do so - so why don't manufacturers "simply" send out devices with different random passwords preset? If the punter loses their password then too bad - or stick it on a label on the item itself. This could all be dealt with in the terms and conditions of sale. Too simple?

    1. Charles 9 Silver badge

      Re: Too simple solution?

      Yes, too simple. People will lock themselves out and your help desk gets hammered. You have to take Stupid into consideration.

      1. Doctor Syntax Silver badge

        Re: Too simple solution?

        "You have to take Stupid into consideration."

        Stupid is the problem. If the punter is too stupid it has to be their problem rather someone else's. I'm a biologist by training. I see no problem in applying Darwinian selection to the IoT.

        How about "Here's your device, there's the password. We have no copy of it. Looking after it is your responsibility."

        1. Charles 9 Silver badge

          Re: Too simple solution?

          "Stupid is the problem. If the punter is too stupid it has to be their problem rather someone else's. I'm a biologist by training. I see no problem in applying Darwinian selection to the IoT.

          How about "Here's your device, there's the password. We have no copy of it. Looking after it is your responsibility.""

          Problem is, Darwinism doesn't jive well with civilized society since it smacks of throwing people to the wolves. Thus attitudes about capital punishment, eugenics, and unwilling euthanasia in its various forms (illness, population, age limits, whatever). There's a reason "Social Darwinism" is considered a dirty word.

  44. Anonymous Coward
    Anonymous Coward

    Bullshit

    "Unlike your PC or your phone, IoT devices don’t have the memory and processing to be secured properly"

    Utter steaming bullshit.

    They're not secured properly because the jackass coders that write their firmware/software don't give two shits about security.

    Apparently, thoughts like "hardcode the password", "yeah just run a telnet server so we can debug it/get the logs easily" and "make sure it can connect back to our servers so we can slurp everything the user does" don't ring any alarm bells.

  45. Anonymous Coward
    Anonymous Coward

    Solution

    There is one solution to IoT problem, taking out vulnerable devices permanently.

    Someone should write a program that searches vulnerable IoT devices and brickes them for good.

  46. Nelbert Noggins

    This isn't even just an IOT problem, the mindset for insecure devices has existing long before IOT.

    Anyone who has hacked around on the average satellite or terrestrial tv box, for example, knows security doesn't come into the design. WTF does everything on a TV receiver need to run as root? This hasn't changed since adding and ethernet port and all the streaming features, telemetry, tablet/phone apps for remote control and casting.

    The consumer, and arguably whole, embedded market is a mess and needs addressing end to end... including the system on a chip SDKs which are buggy and not updated regularly, to the development teams running everything as root with remote access, to the update mechanisms on such devices.

    While the chips are now being put into devices which get internet connected many of the working practices, design and development is still thinking the way it did when they were isolated without any network connection.

    OTA updates or bricking the devices aren't a magic solution, because if the rewards are worth it, the firmware can be captured, examined and flaws found and exploited so they don't trigger alerts. That happens even with devices that have a small group of uses because the manufacturer has stopped supporting the device or they want to add new features the manufacturer won't. Brick the devices and watch US and European companies go bust very quickly as consumers just stop buying devices with internet connections that can use their subscription services.

    For your average consumer knowing which devices are secure and which aren't is impossible to determine. Buying locally isn't any guarantee of security.

    1. Doctor Syntax Silver badge

      "Brick the devices and watch US and European companies go bust very quickly as consumers just stop buying devices with internet connections that can use their subscription services."

      As per my comment above, apply a bit of Darwinian selection. Make it worth while to ship secure stuff. Having sold/issued to the subscriber a steaming pile of ordure isn't an excuse for losing business, it's just a reason.

      In established fields it simply wouldn't be allowed to sell a dangerous design of electrical equipment or vehicle. If it later transpires that something wasn't fit then the vendor will be expected to recall it for remediation; that option should be available to vendors of insecure IoT devices. The the vendor simply goes bust or the customer refuses to accept the recall then there has to be a mechanism for ensuring it's not exposed on the 'net.

      If you want an alternative analogy, consider a contagious disease - of humans or animals. If the disease is sufficiently dangerous TPTB usually have sufficient powers to ensure that humans are isolated and animals destroyed. It's draconian but essential for the wider community.

      1. Charles 9 Silver badge

        "If you want an alternative analogy, consider a contagious disease - of humans or animals. If the disease is sufficiently dangerous TPTB usually have sufficient powers to ensure that humans are isolated and animals destroyed. It's draconian but essential for the wider community."

        But what if the disease is encouraged by a foreign power because it (a) helps to cull their own excesses or (b) some of them are immune, and they don't care about the rest? If everyone but them dies, THEY WIN.

        That's the situation now. Most of this stuff is made in China, and China is noted to be competitive if not hostile to the west, at least economically. In this light, they could care less if the devices are being pwned. Indeed, THEY may be doing the pwning as covert warfare. Meanwhile, they're using channels that are hard to control (alibaba and the rest), AND they can be testy. Not to mention they have nukes AND an Eastern mentality (more accepting of MAD). As the saying goes, it's complicated, and Darwin favors THEM right now. Your move.

  47. Kiwi Silver badge
    Windows

    Hmmm..

    The blame "... lies with the botnet operators – and, perhaps more crucially, the dimwit IoT manufacturers who crank out criminally insecure hardware that can be compromised en masse. Particularly China-based XiongMai Technologies, which produces vulnerable software and hardware used in easily hijacked IP cameras, digital video recorders and network-attached video recorders. These crappy devices were at the core of today's attacks, according to Flashpoint.

    Hrm.. Wonder if, by some slim chance, XMT might just be a lot more aware of the holes, and the botnets that find these holes, than they're letting on? Nah.. Couldn't be that a company deliberately makes their product crap/insecure for $profit_reason...

  48. hypernovasoftware

    "... no one changes their passwords on their gizmos, ..."

    This statement kind of kills El Reg's credibility.

    I do. Thus proving this statement false.

    1. Kiwi Silver badge
      Coat

      "... no one changes their passwords on their gizmos, ..."

      This statement kind of kills El Reg's credibility.

      I do. Thus proving this statement false.

      Are you sure? On every device? Including your car's computer that you don't even know has any form of connectivity? Or those many other devices you have which either have hard-coded passwords or worse, no form of password whatsoever?

      if you're like most modern people, you have devices that you would never even consider having connectivity which not only connect, but connect without ever having had a thought go into their security. Eg any Windows computer.

  49. Nate Amsden Silver badge

    1.2 terabits

    Other news sites are quoting people claiming to be responsible for the attack as throwing 1.2 terabits of traffic in the attack if true would be the largest publicly reported attack in history to date. I wouldn't be surprised if it was even more than 1.2Tbps.

    Honestly as someone who has worked in the industry for almost 20 years(will say again been an enterprise dyn customer for 7 years and have run my own authoritative dns for 20) now it is hard for me to grasp that type of scale in attack form. That is over 1,000 times the size of my org's datacenter uplink to the internet.

    Another way to think about it is many of the current gen 10g 1u rackmount switches have about 1.2tbps of fabric (that is enough for 48x10G and 4x40G running at line rate full duplex within the switch).

    I checked again and dyn reports 18 global locations for their anycast network. I don't know how many servers at each site since dns requests are cheap to process. Obviously they likely process tens to perhaps hundreds of billions of dns requests per day.

    http://dyn.com/dns/network-map/

  50. DropBear Silver badge
    Facepalm

    "Rah Rah Rah IoT!!! Murderkill all the users!!!" That's right, why bother considering how to fix the actual problem - DNS being vulnerable... trolling like there's no tomorrow is easier and so much more satisfying after all.

  51. Velv Silver badge
    Terminator

    "the dimwit IoT manufacturers who crank out criminally insecure hardware"

    Name and shame. Until someone does it there won't be the incentive for the dimwits to fix it.

    1. Charles 9 Silver badge

      Won't work. Many of them are either shameless ("And Proud of It!") or ephemeral ("Are you taking about that company that disappeared last week?").

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019