But the question remains: Should Facebook change the way it monitors users for suicide risk?
‘People need to be aware that … they may be experimented on’
When AI tools flag potential self-harm, those posts go through the same human analysis as posts reported by Facebook users directly.
If AI or another Facebook user flags a post, the company reviews it. If the post is determined as needing immediate intervention, Facebook may work with first responders, such as police departments to send help.
The paper makes the argument that Facebook’s suicide prevention efforts should align with the same standards and ethics as would clinical research, such as requiring review by outside experts and informed consent from people included in the collected data.
“There’s a need for discussion and transparency about innovation in the mental health space in general. I think that there’s a lot of potential for technology to improve suicide prevention, to help with mental health overall, but people need to be aware that these things are happening and, in some ways, they may be experimented on,” Torous said.
“We all agree that we want innovation in suicide prevention. We want new ways to reach people and help people, but we want it done in a way that’s ethical, that’s transparent, that’s collaborative,” he said. “I would argue the average Facebook user may not even realize this is happening. So they’re not even informed about it.”
The Facebook researcher who designed the experiment, Adam D.I. Kramer, said in a post that the research was part of an effort to improve the service — not to upset users. Since then, Facebook has made other efforts to improve its service.
“Suicide prevention experts say that one of the best ways to prevent suicide is for people in distress to hear from friends and family who care about them. Facebook is in a unique position to help because of the friendships people have on our platform — we can connect those in distress with friends and organizations who can offer support,” Antigone Davis, Facebook’s global head of safety, wrote in an email Monday, in response to questions about the new opinion paper.
“Experts also agree that getting people help as fast as possible is crucial — that is why we are using technology to proactively detect content where someone might be expressing thoughts of suicide. We are committed to being more transparent about our suicide prevention efforts,” she said.
Facebook also has noted that using technology to proactively detect content in which someone might be expressing thoughts of suicide does not amount to collecting health data. The technology does not measure overall suicide risk for an individual or anything about a person’s mental health, it says.
What health experts want from tech companies
“It’s another area where private commercial companies are launching programs intended to do good but we’re not sure how trustworthy they are or how private they can keep or are willing to keep the information that they collect, whether it’s Facebook or somebody else,” said Caplan, who was not involved in the paper.
“This leads us to the general question: Are we keeping enough of a regulatory eye on big social media? Even when they’re trying to do something good, it doesn’t mean that they get it right,” he said.
“All these private entities that are primarily not thought of as health care entities or institutions are in position to potentially have a lot of health care information, especially using machine learning techniques,” he said. “At the same time, they’re almost completely outside of the regulatory system that we currently have that exists for addressing those kinds of institutions.”
“This information that they’re gathering — and especially once they’re able to use machine learning to make health care predictions and have health care insight into these people — those are all protected in the clinical realm by things like HIPAA for anybody who’s getting their health care through what’s called a covered entity,” Magnus said.
“But Facebook is not a covered entity, and Amazon is not a covered entity. Google is not a covered entity,” he said. “Hence, they do not have to meet the confidentiality requirements that are in place for the way we address health care information.”
The only protections of privacy that social media users often have are whatever agreements are outlined in the company’s policy paperwork that you sign or “click to agree” with when setting up your account, Magnus said.
“There’s something really weird about implementing, essentially, a public health screening program through these companies that are both outside of these regulatory structures that we talked about and, because they’re outside of that, their research and the algorithms themselves are completely opaque,” he said.
‘The problem is that all of this is so secretive’
“In theory, I would love if we can take advantage of the kind of data that all of these systems are collecting and use it to better care for our patients. That would be awesome. I don’t want that to be a closed book process, though. I want that to be open with outside regulators. … I’d love for there to be some form of informed consent,” Schlozman said.
“The problem is that all of this is so secretive on Facebook’s side, and Facebook is a multimillion-dollar for-profit company. So the possibility of this data being collected and being used for things other than the apparent beneficence that it appears to be for — it’s just hard to ignore that,” he said. “It really feels like they’re kind of transgressing a lot of pre-established ethical boundaries.”