Im honestly curious here about something, if anyone knows more please respond below.
So someone starts uploading dodgy content (dodgy/disgusting/distressing/illegal content). Its flagged by the AI or even by a real person. And the upload is blocked. What happens then? From the sounds of it nothing.
What should happen? Well in my humble opinion, a) account of the uploader blocked, b) the IP address of the uploader recorded, c) if the content is actually illegal (and im guessing most of the truly distressing stuff is whether its because its video of actual criminal activity or just falls foul of various obscentiy laws), then the IP address is passed over to the cops. The police can then obtain necessary warrants, and go arrest these people.
This would result in a few improvements to the anti-social media environment - a) those workers who have to actually peruse all this distressing content would probably suffer less mental illness (if you know that the people uploading this are being brought to justice and that your actions are making the world a better place, then you can withstand a lot more than knowing all you can do is play whack-a-mole without consequence for the bastards), and b) people would be less willing to upload/share this stuff if they were expecting a knock on the door from PC Plod. Everyone wins.
I half expect at least a few commentards to start banging on about "Freedom of Speech" yadda yadda, but two things - a) facebook blocks this stuff already, and there is no requirement for facebook to allow your bollocks on its platform, and b) if the best thing you can say about what your saying is, that its not technically illegal to say it, then you really dont have much of an argument.
Whilst IP Address searches are not going to stop the hardcore wackos uploading stuff, those using VPN's, tunnels, or those coming from countries with no laws on this stuff, etc. the vast majority that reshare these videos are doing it from their own accounts and own residences. Stopping those secondary sources, stops these things spreading beyond their fringe element. And that has to be worthwhile...