You know the old phrase “style over substance?”
Yeah, sometimes ChatGPT is like that.
It’s the guy at the party who’s all swagger and charm, but when you try to pin him down on the specifics of Yemeni foreign policy (or whatever topic he’s bloviating on), it’s clear he doesn’t have a clue what he’s talking about.
Let me give you an example from my own work.
The other day, I was working on a new approach for using AI to evaluate the understandability of online health information. I asked ChatGPT to use an existing health literacy tool to do the assessment.
This should have been my first clue to be skeptical:
I don’t have direct access to the guide page, but I’m familiar with the [tool’s] scoring criteria for patient education materials.
ChatGPT then went on to confidently provide:
- Three overarching categories for evaluation
- 10 specific assessment criteria
- A numbered scoring system
- Interpretation of scores
Y’all, the tool actually has two overarching categories. It has 26 assessment criteria. And the scoring system is yes/no, not numbered.
ChatGPT hallucinated a whole new system instead of saying it didn’t know or needed more information.
The scariest part is that the output was totally plausible stuff. I was very familiar with the health literacy tool, and it still took several queries for me to notice the red flags.
I give you this example not to warn you against using ChatGPT. I’m telling you this so you (and I) can reduce the risk of being bamboozled by it.
Listen, ChatGPT is not trying to fool you on purpose. It doesn’t have that kind of agency. It’s overconfident because the underlying model is trained to be that way:
- It’s predicting text, not retrieving facts. The simple and helpful user interface makes that astonishingly easy to forget.
- The model doesn’t experience uncertainty. It’s trained to produce coherent, authoritative-sounding language.
- It may infer that the “most helpful” response is a complete, confident one, even if what it says is wrong.
So, can you make ChatGPT admit when it doesn’t know something? Kind of. You can use prompts to reduce the chance of hallucinated output. Here are some ideas:
- Explicitly allow for uncertainty.
“If something is unclear or missing, note that explicitly instead of guessing.”
“If you don’t know, respond ‘I don’t know.’”
“If you’re not certain, say so.”
- Ask for sources or evidence.
“List your sources or specify if you’re inferring.”
“What evidence supports that claim?”
“List 2–3 possible answers and indicate your confidence level in each.”
- Force retrieval or verification.
“Based on the latest information from credible web sources, summarize…”
“Here’s the text/document; use only this material to answer. Do not add or infer information that isn’t explicitly present.”
Dazzlingly self-assured answers are ChatGPT’s default. Ask for evidence and invite uncertainty, and you’ll get far fewer overconfident hallucinations.
Real-world example: Show me the data
Let’s compare what happens when we ask ChatGPT for health information with a simple prompt and then with a prompt requiring evidence.
Simple prompt:
What are the top five causes of chronic cough?
ChatGPT says:
My take:
Honestly, this isn’t a terrible answer. ChatGPT answered my question and provided a bit of detail about each of the five causes. But I have no idea where the information came from. It’s plausible, but as we know, that doesn’t mean it’s accurate. I’d have to find my own authoritative sources to fact check before running with it.
Prompt requiring evidence:
According to credible health authorities like CDC or NIH (if known), what are the top five evidence-based causes of chronic cough?
If no authoritative sources are known, respond “uncertain.” Please indicate your confidence in each cause.
ChatGPT says:
My take:
ChatGPT provided the same top five causes with both prompts, but the second answer provided more nuance and highlighted areas of uncertainty. It also gave caveats that would be critical to consider if you were writing about chronic cough; those were entirely missing from the first answer. In the second answer, ChatGPT listed authoritative references that would make it quicker to fact-check.
When writing about health and science, or any other topic involving data or analysis, it can be helpful to force ChatGPT to provide evidence and give authoritative sources. And it’s simple to add that request to any prompt. Give it a go!