back to article AI vuln-hunter bots have seen things you people wouldn't believe

Machine-learning systems are unearthing new classes of bugs in operating systems and apps, according to bods from America's Defense Advanced Research Projects Agency (DARPA). The exact nature of these new bug types remains under wraps, although we hear that at least one involves exploitable vulnerabilities in data queues. …

  1. cbars Bronze badge

    Impressive

    Great stuff.

    Additionally, history has shown that a lot of the technology we use wouldn't exist without the military looking for new weapons. So, swings and roundabouts but hopefully this sort of stuff will eventually filter back to the general populous and make us all safer

    1. Destroy All Monsters Silver badge

      Re: Impressive

      > a lot of the technology we use wouldn't exist without the military looking for new weapons

      I strongly doubt this.

      Actually it's the military who pull in stuff from the civilian sphere, add their own opportunity cost and paint it in olive for no real advantage.

      1. Stevie

        Re: I strongly doubt this

        Ahem. The internet ...

        Also: Concorde.

        Also: Teflon.

        Also: GPS.

        I could go on (atomic power, digital watches, CA adhesives) but I have other things to do today.

        Peaceout.

        1. Captain DaFt

          Re: I strongly doubt this

          "I could go on (atomic power, digital watches, CA adhesives) but I have other things to do today."

          You could have at least mentioned the one that had the biggest influence on modern times: T-shirts!

          1. Version 1.0 Silver badge

            Re: I strongly doubt this

            I thought Digital Watches were a pretty neat idea ...

    2. Yet Another Anonymous coward Silver badge

      Re: Impressive

      The really impressive system was the one which ignored the competition, hacked the conference twitter account and sent messages declaring itself to be the winner to all the news media

  2. amanfromMars 1 Silver badge

    :-)

    This kind of detailed map will enable the researchers to view malware command and control systems, Launchbury said. He didn't elucidate on what other purposes the Department of Defense might use for such technology.

    "Veni, vidi, vici "...... Julius Caesar wannabe type activity springs immediately to mind, although given the global failure of all of the Department of Defence's traditional interventions one cannot be assured or insure against another series of novel catastrophic disasters.

    And talking a Great Game advantage is only so much Bullshit and Ponzi Vapourware if it doesn't deliver Peace and Prosperity to All rather than just an Elite Executive Administration System, and it will create further phantom warriors and virtual terrorists who be wannabe freedom fighters.

    1. Anonymous Coward
      Anonymous Coward

      Re: :-)

      I have to own a piece of those observations given my roles in the military. It's the usual case that someone comes along and gives us a wedgie before we become dead serious about the nature of the next battles/war.

  3. tony2heads

    formally verified microkernel and verified communications software

    When we we see these on consumer systems?

  4. devjoe

    Let's be realistic

    >>One of the competitors found a bug in operating system code used by all players, developed an exploit for the flaw, and used it to attack another competitor to steal some data. A third competitor saw what happened, figured out what the vulnerability was, and patched itself in response.<<

    While the above may be true given some context that is omitted from the article, the paragraph does leave the reader with the impression that the AI automatically discovered a a bug in the OS, developed an exploit for the bug, and developed an attack against a competitor. It also gives the impression that another AI observing this determined that this was in fact what was going on, and that it developed a fix for the bug.

    Neither of those two things are remotely possible in general. And we're not close to this being possible. Analysing software given full knowledge of its sources along with annotations is a difficult problem today, and the best tools are very far away from being able to discover all bugs a code base - we humans can even have difficulty deciding whether a given behaviour is problematic or not.

    Detecting and classifying bugs without full knowledge of the system, simply by observing behaviour, is taking this problem to a whole new level. I am not saying that they did not do this in some isolated case (that the article omits), but it is important to understand that this is not something that can be applied in general. Not today, and not in 10 years either.

    Going from there, to automatically devising a solution to the problem is, yet again, taking this to another level. Yes, I absolutely believe that for a very limited isolated problem this probably can be done - but if people are left with the impression that an AI in general can "fix our software", we're headed for another AI winter. AI's (or anything else for that matter) won't automatically fix our software in any decade soon.

    That said, I'm thrilled that a public competition like this was done and that it gets press - it may help get young people interested in computer science.

    1. slessenberger

      Umm, this is realistic

      Actually you might be surprised at how far this has come since the competition. You should go look at the Cyber Grand Challenge site and the results of it. They have some great videos and commentary about the competition and results. The systems involved actually were given only compiled code and no source. They then analyzed the code, developed exploits and defenses, and attacked each other with no human intervention. That is what made it so amazing.

      The other part of the story is that the software and environment the systems were working in was intentionally limited and isolated. They ran in a reduced environment that was simpler than the software environment that is found on a typical PC or notebook.

      The programs introduced to the systems had been coded with flaws that often were similar to major flaws that have been found in common real world software. In addition to identifying what they were meant to find, the systems also found and exploited errors that the authors did not know were there.

      The surprising thing about the results were that many people thought this level of performance was not possible. Earlier rounds of the competition did not show nearly as much promise but the actual competition provided many surprises to many of us in the industry. It is the speed of progress that is part of the surprise in addition to the results.

      And yes, the systems were able to look at other systems attacks and use that knowledge to patch their own systems and attack others. All this without software or additional knowledge. This behavior is a common thing in attack/defend style cyber CTF competitions played among humans and it is not surprising that the authors of the CRS knew this and incorporated this behavior too.

      There are several companies (for example Trails of Bits, who I am in no way affiliated with but they have a blog post up on the subject called "Automated Code Audit’s First Customer" in which the same principles (even some of the same CRS code) was used on unrestricted real world software for analysis with great success. This stuff is already in the real world and expected to be more mainstream in 2017.

    2. tr1ck5t3r

      Re: Let's be realistic

      If you think about this situation using closed software like MS Windows, I would agree, if you think about this situation with opensource I would disagree.

      WRT to using Open source, you have one AI spotting the problems (Ai1), another AI (Ai2) can rewrite the code randomly, put each iteration up for testing with Ai1. If Ai1 gives it a pass, code is committed. Job Done.

      If you are familiar with coding, how long does it take to change something and recompile it? Perhaps the wrong data type was used for a variable, a String when it should have been a Cstring or Long and the inherent boundary checks datatypes bring to the table.

      Its not hard to do, I even wrote a system in the 00's which rewrote software from procedural toOOP code, converting ISAM to SQL data sources. What the software house thought would take 3 months based on their own programmers experience, took me 1 week to write the convertor, and a few hours for the convertor to run and rewrite the software. Still got paid that 3months though.

      Its just evolution on "steroids" albeit in the silicon domain and not the wider world where we have not even been able to quantify the effect different chemicals have on the long and short decision making process that affect animals including humans.

      Lets face it, I could say something offensive to you whilst you are sober and you wont react, give you some drugs like alcohol which affects your liver and cholesterol production which in turn affects the production of various other hormones that can then go on and affect your judgement, and then you might be able to see how our decision making processes are so widely affected by the chemicals/drugs we choose to ingest, inhale, rub or spray on ourselves.

      Look at this image to see how the liver which produces around 80% of your cholesterol is converted into different hormones. https://en.wikipedia.org/wiki/Cholesterol#/media/File:Steroidogenesis.svg

      Lets not forget the metabolites of alcohol take 14 days to be broken down by the Liver.

      So lets not kid ourselves that we are in control of our emotions or decision making processes, hey?

      Standardised mass market foods go some way to standardise the emotions of humans which in turn helps the legal profession maintain their authority over us by ignoring inconvenient facts like how data you may have been exposed to at some previous point in time coupled with your chemistry at the time can determine future behaviour/reactions which much of main stream science is just scraping the surface on.

      http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0051959

      http://www.lifeextension.com/Magazine/2007/11/report_pregnenolone/Page-01

      "Scientists from the VA Medical Center in St. Louis explored pregnenolone’s benefits in memory retention using a conditioning task in animals that involved learning to avoid a mild electric shock to the foot.37,38 To their surprise, they discovered that pregnenolone displayed beneficial effects in improving memory retention at almost incredibly low doses.1 They also found the response to pregnenolone was much faster than expected had the supplement been working like a typical neurosteroid, leading to the conclusion that something much more dramatic was going on."

      What I take away from this, is that humans with no sex drive is a medical marker to show they are not functioning properly, and considering the weighting we give to old people's influence over our lives, think House of Lords as one example, is society as a whole set up to fail due to the faulty memories and decision making processes of old people with no sex drive, whilst still only allowing alcohol to be dominant recreational drug for society, considering how it affects both short term and long term memories by affecting the cholesterol production and other liver functions?

      Lets not forget Liver is Latin for life.

      Its also interesting how producing so much of the stress hormones can cause ill health by removing/reducing the other hormones which would normally be produced. So whilst stress hormones can increase spatial intelligence as a species are we setting ourselves up for failure in the future, as we know stress hormones affect the ability to remember and learn?

      Studies already show Westerner kids have lower IQ levels.

      Are Western countries suffering the intellectual effects of a baby boomer generation not able to think properly for themselves?

      1. Doctor Syntax Silver badge

        Re: Let's be realistic

        "Its not hard to do, I even wrote a system in the 00's which rewrote software... converting ISAM to SQL data sources."

        Not procedural to OOP, but I saw ISAM to SQL being done in the '80s.

      2. DryBones
        WTF?

        Re: Let's be realistic

        I am not sure if this is an AI bot, a non-English speaker that has a thing for neurology, or just a Brit that has one is the worst cases of textual diarrhea that I have ever seen, but 90% of that was totally irrelevant piss-taking.

  5. Destroy All Monsters Silver badge

    Fat on Hype

    Still interesting to hear about this.

    Now can we please stop shipping shit-tier software for "bring in the trough contents" fighter plane projects or kill off whole villages in Yemen as the "snatch mission" goes south because we are not half as cool and badass as we would like to think we are?

    The saddest part is that when the next "financial" crash comes and some rip the last carpet away from underneath our arses for fun and profit people will say "no-one could have predicted".

  6. Magani
    Boffin

    Have seen things you people wouldn't believe

    Attack ships on fire off the shoulder of Orion?

    C-beams glitter in the dark near the Tannhäuser Gate??

    1. Destroy All Monsters Silver badge
      Windows

      Re: Have seen things you people wouldn't believe

      Actually, I'm seeing Idris (compiles to Haskell) Dependently-Typed Programming Language and this seems to be a good incremental advancement in the state of the art. Fancy AI hype not needed, just theorem proving over the type system.

      Problem is for old curmudgeon deplorables like me who have started with Pascal and never really been in the ML lineage it's nearly Alien Technology.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like