There is a story the legal industry has been telling itself since the very first AI-hallucinated court filing made headlines. The story goes that fabricated case citations are a symptom of carelessness at the margins of the profession, the kind of thing that happens to a solo practitioner buried in work, a small shop without research budgets, or a lawyer who simply did not understand that a chatbot would invent citations with total confidence. It was always a comforting story because it located the problem in someone else, someone less careful, someone with fewer resources. That story just got much harder to tell, because a hallucinated filing has now been tied to Sullivan & Cromwell, one of the most prestigious and deeply resourced firms in the world, and it lands on the same growing pile we track in our hallucinations file.
Sullivan & Cromwell is not a cautionary tale about cut corners. It is a white-shoe institution with a century-plus pedigree, the kind of firm that advises on the largest deals and the highest-stakes litigation on the planet, staffed by graduates of the best law schools and supported by research infrastructure most attorneys can only dream of. If AI fabrication can surface in a filing connected to a firm like that, then the convenient theory that hallucinations are a competence problem at the bottom of the market collapses. This was never about who is careful. It is about how the work actually gets done.
It Is A Workflow Problem, Not A Competence Problem
Here is the uncomfortable reframing the Sullivan & Cromwell episode forces. AI hallucinations do not enter a legal filing because someone is stupid or lazy. They enter because a generative model was used somewhere in the drafting chain, the model produced fluent, authoritative-looking citations that did not correspond to real cases, and the verification step that should have caught the fabrication either did not happen or did not happen thoroughly enough. That failure mode is available to every firm, at every level of prestige, the moment a single attorney or staffer anywhere in the pipeline pastes a model's output into a document without checking every citation against the actual reporter.
Prestige does not immunize a firm against this. If anything, scale increases the surface area. A large firm has hundreds of attorneys, armies of associates, contract lawyers, and support staff, any one of whom can introduce a hallucinated citation into a draft that then moves up a review chain where each reviewer assumes someone below them already verified the cites. The fabrication rides the conveyor belt of delegation and trust right up to the filing, precisely because everyone assumes the prestige of the firm means it cannot be there. It is the same dynamic driving the growing docket of AI lawsuits and sanctions across the profession.
Why Citations Are The Perfect Trap
Legal citations are uniquely vulnerable to this failure because of what they look like. A fabricated citation produced by a language model has the exact form of a real one. It has a plausible case name, a plausible reporter volume, a plausible court, a plausible year. It reads like law because the model was trained on mountains of real law and has learned the shape of a citation perfectly. What it has not done is verify that the specific case it just generated exists, because generating text and checking facts are entirely different operations, and the model only does the first one.
To a human reviewer skimming a long brief, a fake citation is camouflaged among dozens of real ones. The only way to catch it is to actually pull each cited case and confirm it says what the brief claims. That is tedious, time-consuming work, and it is exactly the work that AI was supposed to make unnecessary. The trap is that the tool which generated the fabrication is also the tool everyone hoped would let them skip the step that catches the fabrication. A firm of any size that lets that hope replace the verification step is one careless paste away from the same headline.
What Elite Firms Actually Need To Do
- Mandatory citation verification, no exceptions. Every cited authority in every filing gets pulled and confirmed against the real reporter before submission, regardless of who drafted it or how senior they are. The check cannot be optional or assumed.
- Treat model output as a draft, never a source. Generative AI can be a useful starting point, but nothing it produces enters a filing as fact until a human has independently verified it against an authoritative database.
- Track where AI enters the pipeline. A firm cannot police a tool it pretends nobody is using. Knowing which steps involve generative models is the precondition for verifying their output.
- Stop assuming prestige is a control. The Sullivan & Cromwell episode is proof that reputation and resources do not automatically translate into the one boring procedural step that actually prevents this.
The Lesson The Industry Will Resist
The legal world will want to file this under "even the best firms make mistakes" and move on, because that framing is almost as comforting as the original one about careless solo practitioners. It treats the event as an anomaly rather than a signal. But the signal is clear, and it is the same signal that every AI deployment story keeps sending. A generative model will produce confident, well-formed, completely fabricated content, and the only thing standing between that fabrication and a real-world consequence is a human verification step that is tedious, unglamorous, and the first thing people skip when they trust the tool too much.
The reason a hallucination tied to Sullivan & Cromwell matters more than the dozens of solo-lawyer cases before it is that it removes every excuse. Not enough resources? They have them. Not enough talent? They have it. Not enough at stake to be careful? Their entire business is being careful. If it can happen there, the problem was never the firm. It was the workflow, and every firm that uses these tools without a hard verification gate is running the same risk while telling itself the same comforting story. Keep an eye on the blog, because the next elite-firm version is closer than the profession wants to believe.