Why Gen AI Gets it Wrong

We’ve all heard stories of AI getting critical facts wrong in high-stakes scenarios. From airline deals that don’t exist to court filings with fake citations, it’s common to see AI fall short. Why is it that gen AI can’t get the facts right? Even after multiple rounds of prompting and asking it to “check its sources.” The answer lies in the fundamentals of how large language models (LLMs) work, and a recent memo from Fisher Phillips breaks down some of the core reasons why gen AI goes awry:

“Predictive Generation Without Grounded Facts – As PwC and Forbes note, LLMs aren’t connected to live data unless explicitly integrated. They don’t “know” truth – they just produce what sounds plausible. When asked about unfamiliar or nuanced topics, they may fabricate content in a confident tone.

Poor Prompts or Ambiguous Requests – According to phData, vague, imprecise, or overly broad prompts increase hallucination risk. If you ask for technical, legal, or factual summaries without giving grounding material, the model may make up references or assumptions.

Overuse of “Expert Voice” Prompts – SeniorExecutive.com highlights that asking GenAI to write “as a lawyer” or “as a policy expert” often results in made-up language that sounds more authoritative and is more likely to be trusted, even when the facts are fabricated.

Model Limitations and Data Gaps – IBM and CapTechU explain that hallucinations are more common when the model compresses complex content into short outputs, data is outdated or biased, the user asks about post-training events; and context is too limited to support the task.”

The memo provides a list of tips for preventing AI errors. One highlight is to limit your use of gen AI for high-stakes content. While gen AI might be great for writing internal emails, marketing brainstorming, or preliminary research, it is, at best, only good for a first draft of high-stakes documents. External communications, legal documents, and regulatory filings need human eyes and extensive review. Gen AI might be capable of performing smaller less critical tasks, but leave the consequential documents to the humans. Companies should consider what high-stakes documents they produce and the risk level associated with them. These considerations should heavily factor into company policy about where, and how, gen AI can be used.