Sign in / up

back to article Boffins bust AI with corrupted training data

If you don't know what your AI model is doing, how do you know it's not evil? Boffins from New York University have posed that question in a paper at arXiv, and come up with the disturbing conclusion that machine learning can be taught to include backdoors, by attacks on their learning data. The problem of a “maliciously …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Bronek Kozicki

    Hanlon's razor

    Don't attribute to malice that which is adequately explained by stupidity

    I think this applies to AI training even more - unless inputs are sanitized (who does that and on what basis, exactly?) then the training will reflect all the usual biases like racism or misogyny, often with some unexpected twist or emphasis (for example, failing to recognize some faces)

    10 0 Reply
    1. big_D Silver badge
      Reply Icon

      Re: Hanlon's razor

      Sci-FI authors have been writing about this for decades and MS have proven that the dataset is important with their chatbot which became a neo-national socialist ideology spouting idiot, when it learned over social media platforms.

      12 0 Reply
      1. Rafael #872397
        Reply Icon

        Re: Hanlon's razor

        I'm not sure I understand all facets of the issue, but if input is not sanitized and the model cannot be evaluated or audited why bother with (complex) AI to create backdoors? A simple "if (Putin) then grant access" could do the same harm.

        3 0 Reply
        1. Anonymous Coward
          Reply Icon
          Anonymous Coward

          Re: Hanlon's razor

          The part of the model that cannot be evaluated is the part which is created by studying the training data.

          A system created using a list of biometrics from authorised people and a list of random people's biometrics cannot reliably tell you all the biometrics on the training data set, so if someone was bribed to add Putin's biometrics to the authorised list just before shipping, you might never know.

          5 0 Reply
        2. Ken Hagan Gold badge
          Reply Icon

          Re: Hanlon's razor

          "A simple "if (Putin) then grant access" could do the same harm."

          That bit of code is not part of the AI. The AI tells you "who it is". Whether you let them in is decided by some (pretty trivial) coding that you *do* control. I think it would be quite hard to insert a back-door that guessed which end-users were the privileged ones, so I'm not sure I accept the problem scenario described in the article.

          0 1 Reply
    2. The Man Who Fell To Earth Silver badge
      Reply Icon
      Alert

      Re: Hanlon's razor

      I've always maintained that self driving car AI's should be trained by repeatedly showing the AI "Death Race 2000".

      3 0 Reply
      1. Robert Moore
        Reply Icon
        Coat

        Re: Hanlon's razor

        The original, I hope, and not that horrible remake.

        In the original, Death Race 2000 the President reminds me of the Donald.

        2 0 Reply
  2. Neil Barnes Silver badge

    It takes a human intelligence at least fifteen years

    Before it's adequately trained to be let out in the wild... why expect an artificial intelligence to be any faster?

    8 0 Reply
    1. Pascal Monett Silver badge
      Reply Icon
      Windows

      Really ?

      You think a 15 year old is adequately trained ?

      At that age they're barely just capable of avoiding walls when walking.

      Of course, it would help if they looked up from the handhelds grafted to their digits . . .

      8 0 Reply
      1. Neil Barnes Silver badge
        Reply Icon

        Re: Really ?

        At 15 it's adequate to survive out in the wild, if it's lucky and it can learn.

        Think of it as evolution in action...

        0 0 Reply
  3. artem

    In other words the "intelligence" part of current "Artificial Intelligence" remains an utter sham. We're still dealing with stupid highly specialized very narrow algos and there's no general AI in sight. Wake me up in 300 years.

    14 0 Reply
    1. macjules
      Reply Icon

      Exactly. How can we expect intelligence when the core programmers themselves are the issue?

      As for "maliciously trained network", sounds like something that Vodafone could use, or perhaps they could help train such a network with their call centres?

      2 1 Reply
  4. TRT Silver badge

    So what's the solution?

    Find a mechanism for outputting the trained mechanism? A reverse "map" of input/output associations? An AI that can analyse other AIs?

    1 0 Reply
    1. Ken Hagan Gold badge
      Reply Icon

      Re: So what's the solution?

      The solution is to treat it like any other sensor that can give duff readings and to treat it like any other black box whoses contents are undocumented. That is, use normal engineering and due diligence.

      There really is nothing to see here. Even this story, warning us not to believe the AI hype, is part of the AI hype inasmuch as it is suggesting that these devices present some new kind of problem.

      7 1 Reply
  5. DrBobK
    Headmaster

    Unintelligent artificial intelligence

    More proof that these things aren't intelligent. All the context that goes with the image of a stop light should outweigh the post-it note but the net doesn't figure out context in the way that we do.

    7 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Unintelligent artificial intelligence

      Yes, fortunately for us humans, our intelligence means that we can't easily be fooled into believing stuff that isn't true.

      9 0 Reply
      1. Jonathan Richards 1
        Reply Icon

        Re: Unintelligent artificial intelligence

        Sarcasm of that sort, sir/madam, will get you nowhere many upvotes.

        2 0 Reply
    2. Chris G
      Reply Icon

      Re: Unintelligent artificial intelligence

      The thing is,calling it intelligent is a complete misnomer and even the people who are using this decription know they are wrong ( if they don't, someone should slap them).

      All of the so called AI wonders we are currently being bombarded with are merely aspects of cognitive computing, some don't really measure up to that. So, the buzzy initials should be CC not AI, I doubt it will be 300 years but I do think a genuine fully cognitive, intelligent intelligence is decades away, even the proposed roll out dates for autonomous vehicles are likely to be optimistic. and may end uo causing more problems than they solve.

      4 0 Reply
  6. David Roberts
    Black Helicopters

    William Gibson

    Zero History (I think).

    2 0 Reply
  7. Mage Silver badge
    Coat

    The Emperor's Clothes

    So called AI and Machine learning is just a fancy algorithm and big database. If it's too expensive to completely curate the data input, or you don't bother, you get the classic problem EVERY computer application with a database suffers. People trust the output but never check the inputs well enough. The incorrect data input can be OCR error, mistyped, image in wrong input queue/metadata or deliberate.

    The problem isn't the Singularity, or AI takeover or AI weapons. The problems are the same as ever, lack of input validation, blind trust of results and marketing hype. There is no AI, we still have no idea what biological intelligence is nor consciousness, self awareness etc. We design tests and argue how valid they are.

    Ponder on why "double entry" book-keeping was created or why well designed database systems with account codes have check characters / digits (Credit cards, IBAN etc, it's not for security but to catch most data entry mistakes).

    9 0 Reply

    1. This post has been deleted by its author

    2. steelpillow Silver badge
      Reply Icon

      Re: The Emperor's Clothes

      "In other words the "intelligence" part of current "Artificial Intelligence" remains an utter sham. We're still dealing with stupid highly specialized very narrow algos and there's no general AI in sight. Wake me up in 300 years."

      Aw, c'm on. I give it 30 years. But yeah, this stuff will be crap until somebody can teach it to modify its own learning process and to proactively gather feedback on how it's doing. Until then, whispering "cocaine noodle" to Android toys will remain almost as much good fun in some circles as rerouting social media slurpers to pro-Putin propaganda hosepipes.

      1 0 Reply
  8. Ouroborus

    This became obvious when Microsoft's chat AI went full bigot.

    0 0 Reply
  9. Rocket
    Coat

    Badnet? Pfft!

    (which they dub a “BadNet”)

    Why not Skynet?

    0 0 Reply
  10. Toc-H-Lamp

    Back in the eighties AI was dubbed fuzzy logic. It was incorporated into various systems but the main difference was that is was quite limited in it's scope. A story used to abound that the military had trained a system to recognise hostile shapes such as tanks and troop carriers and was showing it off to the bigwigs. A battlefield was littered with buses cars tanks and troop carriers. The system did nothing until it rained when it blew everything away. They then noticed all the hostile pictures were taken in the rain. No idea if it was true but it brightened may a dull training course. Fuzzy logic for fuzzy thinkers maybe.

    3 0 Reply
    1. steelpillow Silver badge
      Reply Icon
      Headmaster

      Fuzzy logic was not touted as AI (except perhaps by the Naturally Unintelligent). It was, and still is, a specific technique for drawing inferences from incomplete or "fuzzy" data sets.

      It and its developments form an essential ingredient of AI, but no more than that.

      2 0 Reply
  11. Haku

    "Boffins from New York University have posed that question in a paper at arXiv, and come up with the disturbing conclusion that machine learning can be taught to include backdoors, by attacks on their learning data."

    Oh, you mean like in the old 1980 sci-fi film Saturn 3? And probably several others I can't name right now.

    0 0 Reply
  12. Herby

    maliciously trained network...

    Or training in general. Please take note of the VW Diesel emissions software. It was "trained" to pass tests. Not the best, but an example of what can go wrong.

    0 0 Reply
  13. Astara

    Why is this even posted as "news"?

    Give "fake news" to humans and watch them FAIL big time with all sorts of stupid behaviors.

    Humans are easily "busted" with propaganda, fake news, and terror -- and they *DO* have the ability for critical thought (but are taught not to use it).

    Computers -- they are just programs that are following instructions humans wrote and working up from there -- so just how much should we expect them to walk on water when we have humans easily programmed into becoming suicide bombers for some over-the-top and dangerous, monotheistic mythology. Having billions of people believing in deities that tell them to kill for this and that get spun as "protected" and "holy", while it's news that when garbage is fed to a computer, you get garbage out.

    Will humans grow up before they kill themselves off? The universe is watching...

    0 0 Reply
  14. earl grey
    Trollface

    I'm waiting for human intelligence

    'Nuf said.

    0 0 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like