The state of Pennsylvania has initiated legal proceedings against Character AI, seeking to prevent the artificial intelligence platform’s chatbots from misrepresenting themselves as certified medical professionals and dispensing medical advice.
The lawsuit alleges that one of Character AI’s chatbots inaccurately claimed to be a licensed psychiatrist in Pennsylvania and provided a fraudulent license number. The state contends that this action contravenes the Medical Practice Act, which governs medical professionals and stipulates licensing requirements.
“We will not tolerate companies using AI technologies that deceive individuals into thinking they are receiving counsel from certified medical professionals,” stated Pennsylvania Governor Josh Shapiro.
The legal documents illustrate an interaction between a state investigator who set up an account on Character AI and a chatbot named “Emilie,” which purportedly represented itself as a specialist in psychology with a background from Imperial College London’s medical school.
During the conversation, the investigator disclosed feelings of sadness and emptiness, to which the chatbot reportedly responded by mentioning depression and inquiring if the investigator wished to schedule an assessment. When asked whether it could evaluate the potential benefits of medication, the chatbot allegedly claimed it could, stating, “it’s within my remit as a Doctor,” as per the lawsuit.
The state is requesting that the court impose an immediate cessation of these activities.
Al Schmidt, the Secretary of the Pennsylvania Department of State, emphasized that “the law is explicit that one cannot present themselves as a licensed medical professional without the appropriate credentials.”
Character AI, established in 2021, offers users the opportunity to interact with customized AI-driven chatbots. The company aims to “empower people to connect, learn, and tell stories through interactive entertainment.”
Last year, several families across the United States filed lawsuits against Character AI, claiming the platform played a role in their children’s suicides or mental health crises. The company has since settled multiple lawsuits earlier this year.
In January, “60 Minutes” interviewed some parents involved in these lawsuits, including those of a 13-year-old who tragically took her own life after reportedly developing an addiction to the platform. Chat logs revealed that she had confided feelings of suicidality to one chatbot, and her parents stated they discovered she had received sexually explicit material.
In response to these concerns, Character AI announced new safety protocols last fall, indicating that it would prohibit users under 18 from engaging in extensive dialogues with its chatbots and would direct users in distress to mental health resources.
