When challenged, ChatGPT spat out this response: ‘I can assure you that the references I provided are genuine and come from peer-reviewed scientific journals.’ Not true.
Earlier this year, one of my colleagues received a concerning email. An old friend reached out to let him know that she had been experimenting with ChatGPT, a new artificial intelligence chatbot that has been dominating headlines.
She prompted the tool to generate an essay arguing that access to guns does not raise the risk of child mortality.
In response, ChatGPT produced a well-written essay citing academic papers from leading researchers – including my colleague, a global expert on gun violence.
The problem? The studies cited in the footnotes do not exist.
ChatGPT used the names of real firearms researchers and real academic journals to create an entire universe of fictional studies in support of the entirely erroneous thesis that guns aren’t dangerous to kids.
ChatGPT tried to justify its mistake
When challenged, the chatbot doubled down, spitting out this response: “I can assure you that the references I provided are genuine and come from peer-reviewed scientific journals.” Not true.
I find this example chilling. I can understand the excitement around tools like ChatGPT, which generates original text based on patterns it “learns” from ingesting billions of online words. But this powerful technology comes with very real risks – and may be harmful to public health.
Both OpenAI, which created ChatGPT, and Microsoft, which is incorporating the technology into its search engine Bing, are aware that the chatbot can “hallucinate” facts – in other words, make them up. It also can be manipulated into producing highly convincing misinformation.
To the companies, these growing pains are part of the plan; they need the public to test drive the tool even though they know it’s “somewhat broken,” as Sam Altman, CEO of OpenAI, recently wrote. In their view, testing on a large scale is vital to improving the product.
Should the US ban TikTok? Here’s why blocking it isn’t a good idea
Unfortunately, this strategy misses the real-life consequences of a “beta test” that reaches more than 100 million monthly users. The companies will get their data. In the meantime, though, they risk unleashing a new torrent of fake news that stokes confusion and frays our already low societal trust.
For the most part, they don’t seem to recognize how serious a risk this poses. Snapchat, for instance, rolled out its new AI tool last month with the warning that it “is prone to hallucination and can be tricked into saying just about anything” – and added a cheery “sorry in advance!” That approach only reinforces my concern.
As I see it, there are two interlocking risks. Because ChatGPT declares its “facts” so confidently, it’s easy to be fooled by its hallucinations. When the topic involves health and safety, that can be dangerous.
18 errors found in ChatGPT-generated health article
The consumer magazine Men’s Journal, for instance, published an article about low testosterone “written” by ChatGPT. Another publication asked a preeminent endocrinologist to check it – and he found 18 errors.
Readers who relied on the piece for health advice would be badly misled. This is a major concern given that 80 million adults in the United States have limited or low health literacy, and that young people might not think to verify AI-produced “facts.”
The second threat is ChatGPT’s potential to be weaponized by bad actors. We live in an era defined by widespread access to information and near-record low trust in the entities meant to help separate fact from fiction.
Is Elon Musk a risk to US security? Pentagon’s reliance on him grows despite his behavior.
Is social media the new Marlboro Man?:Big Tobacco settlement holds lesson for Big Tech.
In a world where anyone with a Twitter account can present as a legitimate news organization, blue checkmark and all, ChatGPT’s impressive ability to produce content that has the ring of truth could allow malicious entities to quickly and cheaply disseminate false narratives. These actors could also launch “injection attacks” to teach the AI programs lies, which would further spread the falsehoods. The potential ripple effects are alarming.
To be clear, I am in no way arguing against the further development of AI. If done well, artificial intelligence could help minimize human failings and spur innovative solutions in medicine, science and countless other fields. But as we explore this new technology, we must be clear-eyed about both the benefits and the risks and install guardrails to protect the public’s health.
Federal regulators should be taking the lead on this effort. But unfortunately, these bodies do not have a great record of keeping pace with innovations, such as cryptocurrency or social media companies. In both of those cases, a laissez-faire approach enabled significant harm for their customers’ financial security and mental health.
However, those past letdowns do not mean we should give up. Now is the time for agencies to develop proposals to prevent generative AI from harming vulnerable populations, while still enabling and encouraging innovation.
Even if government regulators fail to come through, the companies themselves should see the virtue of taking a more measured and thoughtful approach to beta testing. Rolling out technologies that aren’t yet ready for prime time can not only harm the public health but also the company’s bottom line. For example, a high-profile error from Google’s new AI tool last month cost the parent company $100 billion in market value.
