In a brilliant “a-ha!” moment, it turns out “hallucination-free” AI was a lie the whole time.
Stanford and Yale recently published the first real audit of LexisNexis and Thomson Reuters’ legal AI tools — the ones marketed to every lawyer in America as “100% hallucination-free.”
The results are brutal:
- LexisNexis hallucinates 17% of the time
- Thomson Reuters hallucinates 17% AND refuses to answer 62% of the time
- One response claimed Justice Ginsburg dissented in Obergefell. She did not. She joined the majority.
- Another cited a real case to defend a law the Supreme Court already overturned
- Lexis even cited opinions by “Judge Luther A. Wilgarten” — a judge who has never existed
While I appreciate the progression and immense potential that AI tools like LexisNexis and Thomson Reuters bring to legal studies, this research shows how much we still need the human element in the law profession. AI, no matter how advanced, cannot truly emulate the experience, judgment, and discernment that a seasoned lawyer brings to the table.
Retrieval Augmented Generation (RAG), while valuable, does not kill hallucinations. It merely conceals them behind real-looking citations. And blindly leaning on it is sanctioning the very legal professionals who were originally enticed by its seemingly infallible promise.
This should serve as a stark reminder of the continued significance of human expertise. AI tools serve as aids, and while they can be remarkably efficient, reliable verdicts and advocacy will always require the irreplaceable human touch. They are no substitute for a human lawyer’s skill, experience, and above all, their ability to critically evaluate and discern legal precedent. AI advancement cannot eclipse the need for continuous learning, scrutiny, and skepticism in legal processes.
100% peer-reviewed.
Get new articles from Robin Green delivered directly.
Insights on AI leadership, the future of work, and human collaboration. No noise. Unsubscribe any time.
Subscribe free