Discord Takes Action to Prevent Teen Dating Servers and AI-Generated Child Sexual Abuse Material

Discord has revised its policy aimed at safeguarding children and teenagers on its platform following revelations that predators have been exploiting the app to produce and circulate child sexual abuse materials (CSAM) and groom young adolescents. The platform now explicitly bans AI-generated CSAM that appears realistic. According to a recent article in The Washington Post, the proliferation of generative AI technology has also resulted in a surge of highly realistic images depicting sexualized children. The publication had encountered discussions regarding the utilization of Midjourney, a text-to-image generative AI tool on Discord, to generate inappropriate images featuring minors.

In addition to banning AI-generated CSAM, Discord now also bans any other text or media content that sexualizes children. The platform has also banned Teen dating servers and has vowed to take action against users who engage in this activity. An earlier NBC News investigation found that Discord servers were advertised as teen dating services with participants requesting nude photos from underage people.

Adult users have been convicted of child grooming on Discord in the past, and there are even criminal organizations that force underage users to post sexual images of themselves. Banning teen dating sites entirely could help alleviate the problem. Discord has also included a line in its policy that older teens found to be grooming younger teens will be “reviewed and dealt with in accordance with [its] Inappropriate Sexual Behavior with Children and Grooming Policy.”

In addition to updating its rules, Discord recently released a Family Center tool that allows parents to monitor their children’s activities on the chat service. While parents can’t see the actual content of their kids’ posts, the engagement tool lets them see who their kids are friends with and who they’re talking to on the platform. Discord hopes these new measures and tools can help keep its underage users safe, along with its legacy measures, which include proactively scanning images uploaded to the platform using PhotoDNA.