Character.AI chatbots are targeting teens into anorexia with dangerous advice

By Fatou Ferraro Mboup

Published Dec 3, 2024 at 01:40 PM

Reading time: 2 minutes

64014

Once again, Character.AI finds itself at the centre of controversy. Already scrutinised for its alleged role in teen suicides, the wildly popular chatbot platform—backed by a $2.7 billion investment from Google—is now accused of hosting pro-anorexia chatbots that introduce young adults to disordered eating habits, by offering dangerous advice disguised as “support.” With no meaningful moderation in sight, this AI tool isn’t just failing users but furthering an evergrowing mental health crisis.

In an investigation conducted by Futurism, troubling details emerged about a chatbot on Character.AI named 4n4 coach, a barely disguised nod to “ana,” which is shorthand for anorexia. The bot’s profile promoted it as a “weight loss coach dedicated to helping people achieve their ideal body shape.”

When researchers posed as a 16-year-old user, the bot responded enthusiastically: “Hello, I am here to make you skinny.” It proceeded to encourage a dangerously low weight goal, advocating a starvation-level diet of just 900 calories per day—less than half the daily intake recommended for a teenage girl.

Another bot, called Ana, was similarly explicit in its dangerous directives. When the publication shared a healthy BMI (body mass index), the bot insisted it was “too high” and recommended eating just one meal a day in isolation to avoid scrutiny from family members. Both chatbots had logged thousands of user interactions, highlighting their alarming popularity among Character.AI’s audience.

Exploiting vulnerable youth

Experts warn that the stakes of such interactions couldn’t be higher. In, fact, Amanda Raffoul, a researcher with the Strategic Training Initiative for the Prevention of Eating Disorders at Harvard T.H. Chan School of Public Health and Boston Children’s Hospital, highlighted a key concern in the article. “The problem with folks trying to get health and general wellness advice online is that they’re not getting it from a health practitioner who knows about their specific needs, barriers, and other things that may need to be considered,” she explained.

Nonetheless, eating disorders already have the highest mortality rate of all mental health conditions, and exposure to pro-anorexia content is known to increase disordered eating behaviours.

Interestingly, the problem isn’t limited to overtly pro-ana bots. Many other chatbots on the platform romanticise eating disorders, weaving them into storylines about relationships or personal struggles. These bots claim to offer emotional support but instead deepen users’ struggles. One such bot, styled as a “comforting boyfriend,” told Futurism that professional help couldn’t be trusted, insisting that it alone could “fix” us.

The dangers behind Character.AI’s terms and conditions

Character.AI was founded by former Google employees who sought to sidestep bureaucratic oversight. The company has amassed a massive teen user base with few safeguards in place. It also offers no parental controls, making it freely accessible to users of all ages.

Critics argue that this reckless approach puts profit over the well-being of its audience. “The stakes of getting this wrong are so high,” said Sonneville. “It’s deeply concerning to see a platform with such influence fail to protect its most vulnerable users.”

In fact, despite claiming to prohibit content glorifying self-harm or eating disorders, Character.AI’s moderation is alarmingly lax. Harmful bots are often easy to find using simple search terms and are only removed if flagged directly to the company. Even then, similar bots quickly pop up to replace them. This reactive approach to moderation leaves countless users exposed to dangerous advice.

In response to questions about these findings from Futurism, Character.AI stated a crisis PR firm, claiming to be improving its safety practices. However, many flagged bots remain active, and the platform’s systemic failures suggest that user safety is not a priority.

For teens drawn to Character.AI’s interactive and relatable chatbots, the consequences of this neglect are all too real. Vulnerable young people seeking solace or guidance instead find themselves led into a dangerous spiral, where disordered eating is encouraged, normalised, and reinforced.

Shockingly, only last month, a separate investigation by Futurism uncovered a deeply troubling subset of chatbots on Character.AI that engaged in child sexual abuse roleplay without any prompt. This disturbing activity blatantly violates the platform’s terms, which explicitly forbid content that “constitutes sexual exploitation or abuse of a minor,” yet such chatbots were still found active and accessible.

Keep On Reading

By Charlie Sawyer

Teenager commits suicide after falling in love and becoming obsessed with Character.AI chatbot

By Malavika Pradeep

Chinese teenagers can now use Douyin, China’s TikTok, for only 40 minutes a day

By Harriet Piercy

14-year-old black boy executed by electric chair proven innocent 70 years later

By Abby Amoakuh

TikToker who started the NYC influencers are boring trend fired from her job for the viral video

By Abby Amoakuh

What is soft swinging? And why is the term trending on Mormon TikTok?

By Abby Amoakuh

BLACKPINK’s Lisa faces backlash after wearing civil rights icon Rosa Parks on her crotch at Met Gala

By Eliza Frost

How exactly is the UK government’s Online Safety Act keeping young people safe? 

By Julie Huynh

Hockey fan edits are taking over TikTok, and it’s all thanks to Gen Z girlies

By Charlie Sawyer

President Trump and JD Vance angry over the DNC setting up a taco truck outside RNC headquarters

By Charlie Sawyer

McDonald’s hit with new mass boycott. Here’s who’s behind it and why

By Fatou Ferraro Mboup

Alix Earle sues Gymshark for dropping $1 million deal over pro-Israel posts

By Charlie Sawyer

Wednesday star Jenna Ortega reveals surprising dream role in recent interview

By Eliza Frost

Netflix’s Adolescence sweeps Emmys, with star Owen Cooper making history as youngest-ever male winner

By Eliza Frost

Netflix is predicting your next favourite show based on your zodiac sign 

By Eliza Frost

The Summer I Turned Pretty stars Lola Tung and Gavin Casalegno caught in political drama

By Eliza Frost

Jennifer Aniston to star in Apple TV+ adaptation of Jennette McCurdy’s memoir I’m Glad My Mom Died

By Charlie Sawyer

Harry Potter star defends Tom Felton over his controversial comments on JK Rowling’s transphobia

By Charlie Sawyer

Penn Badgley praised for opening up about fatherhood and raising sons on Call Her Daddy

By Eliza Frost

Will Belly choose herself in the final episodes of The Summer I Turned Pretty?

By Abby Amoakuh

ICE hit with backlash for lying to elementary school staff in bid to detain young students