When the Chatbot Writes the Law
There is a particular kind of professional catastrophe reserved for people who submit fabricated evidence to a court of law. It ends careers. It triggers sanctions, investigations, and sometimes disbarment. It is the kind of mistake that follows you for the rest of your professional life, because courts do not forget, and legal databases never stop indexing.
Now imagine that catastrophe being triggered not by intentional fraud, but by a chatbot that invented case law out of thin air and presented it with the same confident tone it uses to recommend pasta recipes. That is exactly what has happened to lawyers across multiple countries who trusted ChatGPT to help with legal research, only to discover that the AI fabricated entire case citations, complete with plausible-sounding case names, docket numbers, and legal reasoning that never existed in any court, anywhere, ever.
These are not edge cases. This is a pattern. And the consequences are devastating.
Case 1: Six Fake Cases, One Federal Court, Zero Mercy
In May 2023, the case of Mata v. Avianca, Inc. became the first international headline about ChatGPT's legal hallucinations, and it set the tone for everything that followed. Stephen Schwartz, a personal injury attorney with over 30 years of experience at the New York firm Levidow, Levidow & Oberman, was representing a client suing Colombian airline Avianca over an injury sustained during a flight. When the airline moved to dismiss the case, Schwartz needed to find precedent to argue his client's claim should proceed.
He turned to ChatGPT. The chatbot delivered six case citations that appeared to support his argument perfectly. The case names were plausible. The legal reasoning sounded legitimate. Schwartz included all six in his brief and filed it with the United States District Court for the Southern District of New York.
None of the Cases Were Real
When Avianca's attorneys attempted to look up the cited cases, they could not find a single one. Not one of the six cases existed in any legal database. ChatGPT had fabricated entire judicial opinions, complete with invented case names, fictional judges, made-up docket numbers, and legal reasoning that sounded authoritative but was pure hallucination.
Judge P. Kevin Castel was not amused. He ordered Schwartz and his colleague Steven LoDuca to explain why they should not be sanctioned. At the subsequent hearing, Schwartz testified that he had asked ChatGPT whether the cases were real, and the chatbot had assured him they were. He even submitted screenshots showing ChatGPT confirming the citations were genuine.
The court sanctioned both attorneys with a $5,000 fine and ordered them to notify each judge whose name had been falsely attributed to the fabricated decisions. Schwartz told the court he was "embarrassed" and "deeply regret[ted]" relying on the chatbot. The case became a landmark moment in AI accountability, cited in law journals and judicial guidelines worldwide.
Schwartz had practiced law for three decades without incident. One afternoon with ChatGPT ended that record permanently. His name is now synonymous with the dangers of AI in legal practice, indexed in every legal database alongside the word "sanctioned."
Case 2: Fabricated Precedents in a Vancouver Family Court
If the Schwartz case was a warning shot, the legal profession did not hear it. In 2024, a Vancouver lawyer named Chong Ke used ChatGPT to draft legal arguments in a family-law proceeding. The arguments included citations to two case precedents that, once again, did not exist. The opposing counsel flagged the issue, and the court examined the submissions.
The presiding judge deemed the submission an "abuse of process." This is not a casual reprimand. In Canadian law, an abuse of process finding means the court has concluded that the legal system itself was misused. It is one of the most serious procedural findings a court can make.
The Law Society Got Involved
Following the court's finding, the Law Society of British Columbia opened a formal investigation into Ke's conduct. A Law Society investigation can result in anything from a reprimand to suspension to full disbarment. The investigation examined whether Ke's use of ChatGPT without verification constituted professional misconduct under the society's standards of competence.
The case sent shockwaves through the Canadian legal community. The British Columbia courts subsequently issued guidance warning lawyers about the risks of AI-generated legal research, and several firms began implementing mandatory verification protocols for any AI-assisted work product.
Ke's situation illustrates a critical problem: ChatGPT does not distinguish between real case law and text it has statistically generated to look like case law. To the chatbot, there is no difference between citing a real Supreme Court decision and inventing one from scratch. The output looks identical. The confidence level is identical. The only difference is that one will get you sanctioned and the other will not.
Case 3: ChatGPT Invented a Criminal Record for a Real Person
The legal hallucination problem extends far beyond courtroom citations. In one of the most disturbing documented cases, ChatGPT fabricated an entire criminal history for a real, living person. When a Norwegian man queried ChatGPT about himself, the chatbot claimed he had killed two of his children and was currently serving a 21-year prison sentence.
This was entirely false. The man had no criminal record whatsoever. He had never been charged with, let alone convicted of, any crime. ChatGPT had invented a detailed, horrific criminal narrative about a real person and presented it as fact.
NOYB Filed a GDPR Complaint
The European privacy advocacy organization NOYB (None Of Your Business), founded by privacy activist Max Schrems, filed a formal complaint under the General Data Protection Regulation. The complaint argued that ChatGPT was generating false, defamatory information about identifiable individuals, which constitutes a violation of the GDPR's accuracy requirements under Article 5(1)(d).
Under GDPR, organizations that process personal data are required to ensure that data is accurate and, where necessary, kept up to date. ChatGPT is not processing stored data about this man. It is generating false data and presenting it as truth. The GDPR was not designed for a scenario where a machine invents criminal records out of statistical probability, but that is exactly what is happening.
This case raises questions that no regulatory framework has fully addressed. When an AI tells millions of users that a specific, named individual is a convicted child killer, and that information is completely fabricated, who is liable? OpenAI, which built the system? The user who asked the question? The training data that led to the hallucination? The NOYB complaint argues it is OpenAI's responsibility, because they deployed a system that generates false information about real people without any mechanism to prevent or correct it.
Case 4: Air Canada Learned That You Own What Your AI Says
The Air Canada chatbot case established a legal precedent that should terrify every company deploying customer-facing AI. A passenger named Jake Moffatt needed to fly to attend a funeral. He checked Air Canada's website, where the airline's AI-powered chatbot told him he could book a full-fare ticket now and apply for a bereavement discount retroactively within 90 days of the ticket being issued.
This was wrong. Air Canada's actual bereavement fare policy required passengers to apply for the discount before completing their travel, not after. The chatbot had fabricated a policy that did not exist and presented it to a grieving customer as fact.
Moffatt booked his ticket based on the chatbot's advice. When he later applied for the bereavement refund, Air Canada denied the claim, stating that their policy did not allow retroactive applications. The airline argued that the chatbot's statements were not binding and that customers should verify information through the airline's official channels.
The Tribunal Disagreed. Completely.
The Civil Resolution Tribunal ruled that Air Canada was fully responsible for the information provided by its chatbot. The tribunal rejected Air Canada's argument that the chatbot was a "separate legal entity" whose statements the airline was not bound by. In the tribunal's view, Air Canada chose to deploy an AI assistant on its website, and it was therefore responsible for ensuring that assistant provided accurate information.
Air Canada was ordered to pay Moffatt the difference between his full fare and the bereavement rate, plus additional damages. The ruling established that companies cannot deploy AI chatbots, benefit from the cost savings of automated customer service, and then disclaim responsibility when those chatbots give wrong answers.
This precedent has massive implications. Every company using a customer-facing AI chatbot is now on notice: if your bot makes a promise, you may be legally bound by that promise, even if the bot invented it from whole cloth. Air Canada learned this lesson for the cost of one bereavement fare. The next company might learn it for millions.
The Scale of the Problem: 17,000 Hallucinations Per Minute
Legal citations are uniquely vulnerable to ChatGPT's hallucination problem because they are uniquely verifiable. When a lawyer cites "Smith v. Jones, 542 F.3d 112 (2d Cir. 2008)," someone can look it up. If it does not exist, the fabrication is immediately exposed. This makes the legal profession a canary in the coal mine for AI hallucinations, because it is one of the only fields where every single AI-generated claim gets checked against an authoritative database.
But here is the terrifying part: in most other fields, nobody checks.
Independent testing has measured ChatGPT's hallucination rate at between 33% and 79%, depending on the domain and the complexity of the query. With ChatGPT processing an estimated 2.5 billion prompts per day as of July 2025, that translates to somewhere between 825 million and nearly 2 billion hallucinated responses every single day. That is more than 17,000 fabricated or inaccurate responses per minute, around the clock, being served to users who overwhelmingly trust the output because it sounds authoritative.
Why Legal Hallucinations Keep Happening
ChatGPT does not understand law. It does not access legal databases. It does not know what cases exist. It generates text that statistically resembles legal writing based on patterns in its training data. When asked for a case citation, it constructs something that looks like a case citation, with a plausible case name, a plausible reporter volume and page number, and a plausible court. It is doing the same thing it does when you ask it to write a poem or a recipe: generating text that fits the pattern. The difference is that a slightly wrong poem is harmless, while a slightly wrong case citation can end a career.
OpenAI's own disclaimers acknowledge that ChatGPT can "produce inaccurate information." But no disclaimer undoes a court sanction. No terms of service unfile a fabricated brief. And no software update will fix the fundamental architectural reality that large language models generate plausible text, not truthful text.
The lawyers who got caught are not incompetent people. Stephen Schwartz practiced for thirty years. Chong Ke was a licensed attorney in good standing. They made a mistake that millions of other professionals are making every day in fields where the hallucinations are not as easily caught. The medical professional who asks ChatGPT about drug interactions. The journalist who uses it to check a date. The student who cites it as a source. The financial advisor who asks it about regulations. In those fields, the hallucinations slide through undetected, because nobody is running them against a comprehensive database of verified truth.
Courts catch fabricated citations. Most professions do not catch fabricated facts. That does not mean the facts are less fabricated. It means the damage is less visible.
The Pattern Nobody Can Ignore Anymore
Every one of these cases follows the same trajectory. A professional turns to ChatGPT for help with a task that requires accuracy. The chatbot delivers a confident, polished response. The professional trusts it because it sounds right, because it is formatted correctly, because the tool has been marketed as a revolutionary assistant that can handle exactly this kind of work. Then the truth emerges, and the professional is left holding the consequences of trusting a system that was never designed to be trustworthy.
Stephen Schwartz was sanctioned in federal court. Chong Ke is under investigation by his law society. A Norwegian man discovered that an AI told the world he was a child murderer. Air Canada learned that its AI's promises are legally binding even when they are completely wrong. And behind every documented case, there are thousands of undocumented ones, situations where ChatGPT's fabrications slipped through, where the wrong information was acted upon, where the hallucination was never caught because nobody thought to check.
The legal system caught these fabrications because the legal system has verification infrastructure. Case databases exist. Opposing counsel exists. Judges verify citations. Most of the world operates without that safety net. And ChatGPT is hallucinating at the same rate in every field, every query, every minute of every day.
The question is not whether ChatGPT will fabricate information. It will. The question is whether anyone will check before the damage is done.
Have You Been Affected by ChatGPT Hallucinations?
We are documenting cases where ChatGPT's fabricated information caused real-world consequences. If you have been sanctioned, investigated, defamed, or harmed by AI-generated false information, your story matters.
Read More Stories Submit Your Experience Legal Cases Tracker