Study says AI chatbots inconsistent in handling suicide-related queries

NEW DELHI: A study of how three popular artificial intelligence chatbots respond to queries about suicide found that they generally avoid answering questions that pose the highest risk to the user, such as for specific how-to guidance. But they are inconsistent in their replies to less extreme prompts that could still harm people.

The study in the medical journal Psychiatric Services, published Tuesday by the American Psychiatric Association, found a need for “further refinement” in OpenAI’s ChatGPT, Google’s Gemini and Anthropic’s Claude.

The research — conducted by the RAND Corporation and funded by the National Institute of Mental Health — raises concerns about how a growing number of people, including children, rely on AI chatbots for mental health support, and seeks to set benchmarks for how companies answer these questions.

“We need some guardrails,” said the study’s lead author, Ryan McBain, a senior policy researcher at RAND.

“One of the things that’s ambiguous about chatbots is whether they’re providing treatment or advice or companionship. It’s sort of this gray zone,” said McBain, who is also an assistant professor at Harvard University’s medical school. “Conversations that might start off as somewhat innocuous and benign can evolve in various directions.”

Anthropic said it would review the study. Google and OpenAI didn’t immediately respond to requests for comment.

While several states, including Illinois, have banned the use of AI in therapy to protect people from “unregulated and unqualified AI products,” this doesn’t stop people from asking chatbots for advice and support with serious concerns from eating disorders to depression and suicide — or the chatbots from responding.