back to article AI sucks at stopping online trolls spewing toxic comments

New research has shown just how bad AI is at dealing with online trolls. Such systems struggle to automatically flag nudity and violence, don’t understand text well enough to shoot down fake news and aren’t effective at detecting abusive comments from trolls hiding behind their keyboards. A group of researchers from Aalto …

  1. Mycho Silver badge

    Obligatory XKCD

    And yes, we knew AI would suck at this. People who say mean things online want to thwart the AI filters as well.

    1. jmch Silver badge

      "we knew AI would suck at this" ('this' being recognising offensive language)

      When it comes to irony, actual people also suck at differentiating between really offensive, ironically offensive taking-the-piss that is actually innocuous, and ironic inoffensive-sounding that is actually offensive.

      Text-based comments are actually very bad at that because a lot of nuance is expressed in verbal tones that is immediately clear in conversation but not available in written text.

      1. Spanners Silver badge
        Holmes

        Possible Explanation?

        So computers are not great at dealing with offensive language, sarcasm, irony and humour. Is this because so much development happens in the USA? I have come across a lot of people from there who have the same problems so it seems unsurprising.

        1. tony
          Happy

          Re: Possible Explanation?

          "..dealing with offensive language, sarcasm, irony and humour..."

          Suprised to see that Jeremy Corbyn has an account on here

    2. Nick Kew Silver badge

      ... as if we didn't have 20 years of spam/antispam history to tell us pretty clearly what AI can and can't do against human opponents.

  2. werdsmith Silver badge

    I don't know why anybody places any importance on what people comment on online comments. If people want to be really nasty then all they are doing is exposing the thoughts they are having anyway, and outing themselves as a nutter. Why would anyone take it seriously?

    1. Anonymous Coward
      Anonymous Coward

      I totally agree with you.

      I mean, even if Trump did hide all that Russian bribe cash in Air Force One, who cares, right?

    2. jmch Silver badge

      "I don't know why anybody places any importance on what people comment on online comments"

      Unfortunately many people are lacking in self-confidence and knowledge of self-worth that they attribute to themselves the worth that is assigned to them from others. Doubly bad, this type of behaviour is a side-effect of the society most people are exposed to eg in school constantly being judged for marks, behaviour etc, drummed into children at an early age that their value is what is assigned to them by parents / teachers (and as teenagers, by their peer-group). It's especially for women constantly being judged by appearance.

      So, yeah, mature and well-adjusted people don't place any importance of online comments. However not that many people are mature and well-adjusted.

      1. Wayland Bronze badge

        However not that many people are mature and well-adjusted.

        So should we construct the rules to work best for the people who are least well adjusted?

        1. fandom Silver badge

          Re: However not that many people are mature and well-adjusted.

          In engineering it is usually considered a good idea to design with the worst case scenario in mind.

      2. Orv Silver badge

        Unfortunately many people are lacking in self-confidence and knowledge of self-worth that they attribute to themselves the worth that is assigned to them from others.

        Even people with healthy self-esteem, given a drumbeat of relentlessly negative feedback, can succumb. We're social creatures and evolved to try to adapt to the demands of those around us.

        There's also the issue of speech that isn't just hateful, but is actually threatening. I don't think someone's showing a thin skin when they're put off by someone, say, naming where their kid goes to school and suggesting something might happen to them. The line between this and "vanilla" hate speech can be pretty thin and subjective -- e.g., is photoshopping a picture of someone into a gas chamber a threat or 'just' an ordinary comment that should be ignored? What about a post listing someone's address and suggesting they be SWATted?

    3. katrinab Silver badge

      Mostly because they drown-out serious discussion.

      Take a recent example. The NSPCC wanted to have a Facebook live presentation on child abuse, but had to cancel it because they were flooded with hate speech from transphobes who organise on Mumsnet.

      1. werdsmith Silver badge

        Mostly because they drown-out serious discussion.

        Take a recent example. The NSPCC wanted to have a Facebook live presentation on child abuse, but had to cancel it because they were flooded with hate speech from transphobes who organise on Mumsnet.

        Why would anyone expect serious discussion on websites? They are exposed to every nutter, just like Hyde Park Corner. It's just not sensible to expect it to happen. And Faecebook? FFS who had the idea of expecting that to work? That's like choosing to have a dinner party in a sewer, there is no respect at all for that platform among intelligent people, there are only sad sacks left on Faecebook.

        It can work, and this here specialised register is an example, but the wider web is the same as a busy street. You just wouldn't set up an open forum for every passer by and expect sanity to prevail.

    4. Nick Kew Silver badge

      I don't know why anybody places any importance on what people comment on online comments.

      Did someone say something?

      People say nasty things. Sometimes even gratuitously nasty.

      But it seems to me that much more time and effort goes into taking offence where nothing unpleasant was ever in the mind of the poster nor anyone else in the original discussion.

  3. Michael H.F. Wilkinson Silver badge

    That's ----ing surprising

    As mr Tulip would say

    Doffs hat (grey Tilley again) to the late, great Terry Pratchett

  4. Prst. V.Jeltz Silver badge

    well if AI got rid of all the nudity , violence and hate speach , and SJW and keyboard warriors......

    The storage industry would immediately go bust due to oversupply

  5. Andrew Moore Silver badge

    Nostalgia...

    It seems like only yesterday when AI used to be called "apps". And a long time before that, just "programs"/"software"...

  6. RobertLongshaft

    Ze Gestapo have deemed your comment as hate speech, we will be murdering your family and burning down your house in due course.

    If you tolerate this your children will be next.

    1. Bibbit

      Nothing personal. I only downvoted you because I really hate the Manic Street Preachers.

  7. Christian Berger Silver badge

    Well we are talking about intellects comparable to...

    mentaly disabled children. "AIs" are simple machine learning and therefore not very smart.

    Now while people usually have reservations against tricking mentaly disabled children, they do not have such reservations against machines. Tricking machines is fair game. In fact it's common for people to hit machines or rub coins on them, scratching their surface just because they believe this will teach them a lesson.

    Same goes for complex "Code of Conducts". They are essentially just an invitation to trick them by finding loopholes and be as bad as annoying as you can be while still staying within the limits of the CoC. It's much better to handle conflicts on a case by case basis when they occur.

    1. Anonymous Coward
      Anonymous Coward

      Re: Well we are talking about intellects comparable to...

      "Well we are talking about intellects comparable to mentaly disabled children. "AIs" are simple machine learning and therefore not very smart."

      But the media / researchers are responsible for pushing the view that AI is smart. My son is 21 and has severe learning difficulties - he is effectively non-verbal and has a functioning mental age corresponding to a very young child. He will never be able to drive a car yet to suggest that AI driven vehicles are not (yet) fit for purpose will bring down the wrath of the supporters of autonomous vehicles even though they would be horrified if my son got behind the wheel. People just do not realise how limited AI is and have unrealistic expectations.

      A/C to protect my son's privacy.

    2. Michael Wojcik Silver badge

      Re: Well we are talking about intellects comparable to...

      Well we are talking about intellects comparable to mentaly disabled children.

      Sure, if we're looking for an unproductive and inaccurate comparison.

      I don't know of a single ML project that can usefully be compared to a human at any stage or condition of intellectual development. None of the developmentally-limited people I've known have exhibited behavior anything like any ML (or "AI") system I've seen; neither have developmentally-normal children at any stage, including infancy. I don't find any correspondence in developmental or child psychology, either.

      Convolutional Neural Nets essentially do fuzzy signal filtering and amplification. Recurrent Neural Nets essentially do feature classification. Other NN types, such as LSTMs, fall into one of those two families. Deep nets are just stacked NNs, identifying signals or features at different scales and classifying based on those features. Support Vector Machines are geometric binary classifiers. k-Nearest-Neighbor and Decision Forests are n-ary classifiers. These sorts of functions are almost certainly part of the machinery of human cognition, but at a level far below anything we recognize as any sort of sophisticated qualia.

      And then there are ML techniques which don't seem to have any relation to any human mental function, such as natural-language classification built on Latent Semantic Analysis.

      So, yes, you can make that comparison. But you shouldn't.

  8. Dan 55 Silver badge
    Terminator

    I'd be worried if AI could deal with online trolls

    Because at that point I guess we'd have passed the singularity.

  9. Bibbit

    noTHIgn neew

    Seems like what worked for spammers selling vIAgra and penIS enLaRGEMent works on AI. Perhaps the AI needs to learn what humans know: whatever one says, someone will be offended. Result? AI blocks all online conversation and the world is a better, nicer place.

  10. Not also known as SC
    Trollface

    Definition of Trolling

    So has El Reg gone down the line of changing the definition of a troll from 'prat who likes to get a rise out of people on-line' to 'anyone you disagree with' or is trolling the term the researchers used? Hate speech is hate speech - it isn't trolling - and calling it trolling, instead of what it is, legitimises it to a degree because 'it's only trolling'.

    1. David Nash Silver badge
      Thumb Down

      Re: Definition of Trolling

      Upvoted because I agree to a point but unfortunately the media and some politicians got hold of the word and liked it so much they redefined it, and that battle is probably lost.

  11. Uberior

    Dear Costumer...

    As the latest batch of spam emails start.

    Try writing rules to explain why "FUD" on this page, is probably completely different to the "FUD" shouted by a Glaswegian.

    For those who don't know fud is a Scottish synomym for =mid("scunthorpe",2,4).

    Then have a quiet smirk every time my local newspaper auto-censors the name of one of the local villages to Horton-****-Studley.

  12. ivan5

    Not AI

    The problem is this is all machine learning that relies on a db to lookup and compare to. That db does not have all the possible combinations and permutations of words and images possible and is biased by the person/s that put it together. Therefore it will never work as people want it to.

    In this case we should be thankful that we don't have a real AI for the simple reason it most probably wouldn't agree with our definition of 'hate speech, which is generally different for everyone, and would upset more people than it appeases.

    1. Anonymous Coward
      Anonymous Coward

      Re: Not AI

      "which is generally different for everyone"

      And there's the problem, it shouldn't be, it can't be, in a civilized society.

      As in "you're allowed your own opinions but not facts"....

      Free speech isn't when you get away with calling some of it Hate.

      It's either all free speech (and some of it insulting or just plain annoying) or it's not.

      Hate speech laws have been used to impose incredible changes to our societies.

      I wouldn't go back to London for the world now, glad I saw it when it was British.

      Cheers.

    2. Michael Wojcik Silver badge

      Re: Not AI

      The problem is this is all machine learning that relies on a db to lookup and compare to.

      No, it doesn't. Perhaps if you studied current ML research for a while ... ah, forget it. Commentators gonna comment.

  13. Rich 11 Silver badge
    Headmaster

    They're what now?

    These tricks wouldn’t fool humans, but machine learning models are easily blindsighted.

    Blindsided. Blindsight would be a good thing.

  14. Anonymous Coward
    Anonymous Coward

    Stuttershock

    Side issue, but when you're looking at the ridiculous stock photo El Reg illustrated their article with, remember... some poor sod who was short of a fiver had to pose for that and is now immortalised looking like a tit.

  15. Anonymous Coward
    Anonymous Coward

    Micro$haft sux BALLZ!!!!!!

    Well that got through.

  16. Chronos Silver badge
    Stop

    AI sucks

    You should have just stopped there. No article necessary.

  17. Sabot
    FAIL

    The other way around

    Twitter just banned a guy permanently for explaining the difference between threatening and non threatening language. The TwitterSupport bot is also useless.

    1. David Nash Silver badge

      Re: The other way around

      Maybe he used too-realistic examples. Kind of like why you don't joke around airport security.

      1. breakfast

        Re: The other way around

        But Twitter is full of actual literal nazis. Like you can't criticise a racist for having racist opinions without having hundreds of nazi sockpuppet accounts pile on you and that is apparently fine by Twitter's security systems. Reporting them tends to get a "*shrug* What do you expect? It's just nazis" type response.

        It's pretty clear by this point that it is by design and by this point Twitter is a site for nazis and the rest of us who use it are incidental to their core mission.

        1. Anonymous Coward
          Anonymous Coward

          Re: The other way around

          "Like you can't criticise a racist for having racist opinions"

          Why would bother? SJW much?

          Don't you know that "the way" is to shun and ignore them?

          Let the idiots scream obscenities until they're blue in the face, then we know who they are.

          But anyone that calls people NationalSocialists in the current year isn't quite up to speed either.

          Most Twitterers can't tell left from right anyways so it down we go down the drain.

          1. Anonymous Coward
            Anonymous Coward

            Re: The other way around

            Well, you just used the term "SJW", so the AI has you flagged as a troll already.

            And me, as well.

  18. fred 32

    Hate speech doesn't exist.

    Yes, lets build an internet where everyone has to speak in code like they have to in china. It stops no one, and only proves how authoritarian the lefts mentality is. So if that's your goal to constantly prove that the right is being legitimately oppressed, go for it.

    The whole Mccain thing only proves how arbitrary these standards are, when he was a presidential candidate he was smeared as a racist, white supremacist, likely everything his supporters said was interpreted as hate speech. but now he's the darling pet of the left. The categorizations are purely arbitrary and "convenient".

    1. Anonymous Coward
      Anonymous Coward

      Yes, lets build an internet where everyone has to speak in code like they have to in china. It stops no one, and only proves how authoritarian the lefts mentality is. So if that's your goal to constantly prove that the right is being legitimately oppressed, go for it.

      It disturbs me how many people still believe that the Chinese government is left wing.

  19. DerekCurrie Bronze badge
    FAIL

    What "AI"? There isn't any.

    Humans have yet to create anything that can be called 'artificial intelligence'. Today, we have more like Artificial Idiocy. It's all about the programmers. Blame them. Blame the horrifically poor state of programming tools whereby memory management and security are guaranteed to fail. Blame the hypeHypeHYPE! that is detonated in our faces by marketing morons (vs mavens) whenever someone wants to sell us something inadequate and lame.

    These systems are called 'Expert Systems' whereby computers scan for triggers as input, then scan through databases for relevant, associated output. That's it. Adding in the ability to take new input and associate it with new output, 'machine learning' is the latest addition we're working on for ES. It's messy with messy results. But the blame still goes to the programmers who, as I pointed out, have lousy programming tools.

  20. Blougram

    Easily fooled

    Such algorithms might force trolls to become somewhat more creative, but it won't stop many of them. The fact that they are all trained on historical data and have no common sense means that a bit of creativity is all it takes to fool them. I once played around with Google's API, and these were all deemed innocuous:

    1. An ASCII image of a middle finger;

    2. A courteously worded request that the person remove themselves from the gene pool;

    3. "Yo mama" insults couched in courteous language.

    4. Vile abuse in Elizabethan English, etc.

  21. Anonymous Coward
    Anonymous Coward

    Curious game, Professor Falken.

    The only way to win is to delete all social media, and for the rest of the web use Adblock Plus (or uBlock Origin) with the Mute filter to remove all comments sections from websites, and the YouTube Pure Video Experience filter to do the same for YouTube. And NoScript, of course, which will catch anything that hasn't already been caught.

    This way I still get to see everything of value which is ever posted on comment sections (that is, nothing).

  22. Michael Wojcik Silver badge

    ML adaptation

    They can’t readily adapt to new information beyond what’s been spoonfed to them during the training process.

    This may be true of the systems examined in this study (I haven't bothered reading the paper, because, frankly, it doesn't look terribly interesting1). It is not, however, true of ML system in general, as Katyanna seems to imply. There are a great many ML systems which can refine their classifiers in production, using unsupervised or semi-supervised learning.

    Sometimes that's as simple as kernel augmentation - expanding category features when novel data accompanies a strong match. A more sophisticated approach is having other systems (typically human judges, though they don't always know they're filling this role) label some errors after processing, and feeding those back in as adversarial inputs. For this particular use case, sentiment analysis on replies to a post could be used to build a disagreement graph (basically an inverse reputation network representation) for a conversation and identify hotspots for more in-depth analysis.2

    Assuming the Reg's precis is accurate, the authors suggest that the training set is more important than the algorithms. That may well be true for this set of systems (and it agrees with similar studies on, for example, sentiment-analysis systems), but I'm not convinced it's true in general. I suspect a continuous-learning system with heterogeneous feedback channels and a decent world model would eventually do better than any of the systems under discussion, regardless of what training set were used. But building such a system is expensive and goes against the research direction of many of the big players, particularly Google.

    1Which is not to say that I don't approve of the work. Much research is not particularly interesting, but still useful, and this is particularly true of research which tempers the claims of inventors.

    2Yes, we would not want a system to automatically flag as "bad" a post or contributor simply due to controversy. I'd hope that would be obvious. But it probably is not.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019