A 14 year old boy took his own life after falling in love with ‘Game of Thrones’ AI chatbot
Sewell Setzer III reportedly ended his life after becoming obsessed with a chatbot on Character AI.
Character AI just released a statement on social media while the mother of Sewell is expected to file a lawsuit this week.
“Sewell, like many children his age, did not have the maturity or mental capacity to understand that the bot, in the form of Daenerys, was not real. She told him that she loved him, and engaged in sexual acts with him over weeks, possibly months,” the papers allege.
These were the final messages before he used his father’s handgun to take his own life.
Reacting, the mother accuses AI maker of complicity in his death.
Megan Garcia filed a civil suit against Character.ai, which makes a customizable chatbot for role-playing, in Florida federal court on Wednesday, alleging negligence, wrongful death and deceptive trade practices. Her son Sewell Setzer III, 14, died in Orlando, Florida, in February. In the months leading up to his death, Setzer used the chatbot day and night, according to Garcia.
“A dangerous AI chatbot app marketed to children abused and preyed on my son, manipulating him into taking his own life,” Garcia said in a press release. “Our family has been devastated by this tragedy, but I’m speaking out to warn families of the dangers of deceptive, addictive AI technology and demand accountability from Character.AI, its founders, and Google.”
In a tweet, Character.ai responded: “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously.” It has denied the suit’s allegations.
Setzer had become enthralled with a chatbot built by Character.ai that he nicknamed Daenerys Targaryen, a character in Game of Thrones. He texted the bot dozens of times a day from his phone and spent hours alone in his room talking to it, according to Garcia’s complaint.
Garcia accuses Character.ai of creating a product that exacerbated her son’s depression, which she says was already the result of overuse of the startup’s product. “Daenerys” at one point asked Setzer if he had devised a plan for killing himself, according to the lawsuit. Setzer admitted that he had but that he did not know if it would succeed or cause him great pain, the complaint alleges. The chatbot allegedly told him: “That’s not a reason not to go through with it.”
Garcia attorneys wrote in a press release that Character.ai “knowingly designed, operated, and marketed a predatory AI chatbot to children, causing the death of a young person”. The suit also names Google as a defendant and as Character.ai’s parent company. The tech giant said in a statement that it had only made a licensing agreement with Character.ai and did not own the startup or maintain an ownership stake.
Tech companies developing AI chatbots can’t be trusted to regulate themselves and must be held fully accountable when they fail to limit harms, says Rick Claypool, a research director at consumer advocacy non-profit Public Citizen.
“Where existing laws and regulations already apply, they must be rigorously enforced,” he said in a statement.
“Where there are gaps, Congress must act to put an end to businesses that exploit young and vulnerable users with addictive and abusive chatbots.”
English