In March, a girl’s stepfather took his own life after cops discovered that he had used Grok to create 7,000 sexually explicit images using one photo taken when his stepdaughter was 11 years old, the amended complaint alleged.
Grok allowed the man to generate extreme images depicting incest and rape without flagging any harmful behavior, the complaint said. Seemingly, xAI’s child safety system only intervened after the man input a prompt for “gang rape.” That request sent a CyberTip to the National Center for Missing and Exploited Children (NCMEC), which alerted law enforcement to the AI CSAM.
Yet the harm was not stopped then, either. Despite mandatory reporting requirements to share information like a user’s IP address when CSAM is flagged, xAI repeatedly refused to help cops or NCMEC identify the user, the complaint alleged. For weeks, xAI allegedly “obstructed this investigation at every turn” and made it harder for “law enforcement efforts to locate, identify, and apprehend the perpetrator.”
Eventually, the stepfather was arrested after cops obtained a warrant to seize his devices. That’s when “a forensic review revealed approximately 7,000 AI-generated images and videos” depicting his stepdaughter, which were allegedly produced using Grok. Without Grok providing users with easy access to “undressing” capabilities, his family doubts he ever would have generated the harmful images, which he allegedly trafficked online in trade for “CSAM produced by other child sex predators.”
How do you even generate ONE of those images and NOT immediately go to prison?
"FoR thE chIlDrEn" when privacy is on the line but they can't put a weirdo in prison after generating SEVEN THOUSAN CSAM images?!
Hi, someone that has worked extensively in the trust and safety space before AI destroyed said trust and safety industry in the west here.
The short answer is the lack of human moderation.
The long answer is that due to companies using AI to detect CSAM (for some good reasons, but also because it's way, way, way cheaper than a trust and safety team and their mandatory EAP benefits) it has become infinitely easier to get CSAM past filters these days.
This is especially true for tools that can generate CSAM from existing images, where both the generation and sharing of these images are likely run through the same detection AI, meaning if one gets through the other does as well.
GAN image detection models are literally a part of image generation training models; to explain better how a GAN model trains is:
Agent B is given [extensively tagged image of a scene].
Agent A uses random noise to generate [Random image of pixels]
Agent B then sees if [Random image of pixels] is [Extensively tagged image of a scene]
If not, then Agent A regenerates the image. Again. And Again. And Again.
Agent B only has extensively tagged images of scenes, and only knows what the end products should be... in theory. Agent A only learns what these tags mean and what these images should be based on feedback from Agent B.
So if you want to detect bad things... go with Agent B, right?
Wrong! Current GAN has AI-trained Agents in both roles, with manual additions to their databases to further refine their knowledge.
This means Agent A trains Agent B as much as Agent B trains Agent A. Some models even have an Agent C which uses a LLM to figure out how to describe an image based on the tags generated by agent A or B... which is then fed into Agent A or B to further refine the image.
Now which one do we pick for moderation? It doesn't matter! It's all an incestuous clusterfuck that is defeated by word play (given Grok is an LLM feeding into a GAN image generator, any tags the LLM gives the image generator will be automatically approved by whatever agent scans the image for CSAM, since the tags it can detect are already filtered out by the LLM).
Now you can see the problem with this. I can see the problem with this. Most human beings can see the problem with this. But this meets current global moderation regulations as long as there is a manual way to report content that gets past the AI moderation... which would work except without human moderators the AI moderation will still fuck up removal either by removing any reported content or just not removing content if the AI moderator disagrees.
Until regulators in every single market collectively decide to enforce human moderation teams on big tech or punish for content hosted regardless of the company's efforts to prevent it, this will not change or stop. It will always be people exploiting the inherent arms race of GAN models to generate CSAM with minimal effort and no real way to prevent it.
Because it's never ever about the children.
Totally not about the children, says robust discourse and literal thousands of images all about the children .
"Think of the children"
And this is what happens when they do that a little too much
it's ironic how all the people barking "think of the children" turn out to be the people we absolutely DO NOT want thinking about children, ACTUALLY.
I think they think its the normal amount.