Why Legal AI Gets It Wrong (And Why That’s Not the Real Problem)
A practical guide for Australian and New Zealand lawyers on working with AI the same way you work with people
In August 2025, a KC stood before Justice James Elliott in the Victorian Supreme Court and apologised. His defence team had filed submissions in a murder trial containing fabricated quotes from parliamentary speeches and case citations that did not exist. The errors were only discovered when the judge’s associates could not locate the cases and asked for copies. The lawyers admitted they had “checked the initial citations and assumed the others would also be correct.”
A month later, a Victorian solicitor became the first Australian lawyer formally sanctioned for AI-related misconduct. He had submitted documents to the Federal Circuit and Family Court containing entirely fabricated citations generated by AI software. He did not verify them. He is now stripped of his right to practise as a principal, cannot operate his own firm, and must work under supervision for two years with quarterly reporting to the regulator.
These are not isolated incidents. Since 2024, more than twenty similar cases have surfaced in Australian courts. A Melbourne firm was ordered to pay costs after citing fake cases. A Western Australian lawyer was referred to the state regulator. A Victorian defence lawyer in a child murder case submitted non-existent citations and fabricated quotes.
The pattern is always the same: lawyer uses AI, lawyer does not verify output, lawyer submits fabricated material to court, lawyer faces consequences.
But here is what frustrates me about how these stories get reported: the headlines scream about AI as if the technology is the villain. The actual failure in every single case was a lawyer abandoning basic professional practice. They would have faced identical consequences if they had trusted a paralegal who invented cases, relied on a colleague’s faulty memory, or cited headnotes without reading judgments.
The tool did not fail them. They failed to practise law as normal.
What This Article Is Actually About
If you clicked on this article expecting another breathless warning about how AI will destroy your career, you are in the wrong place.
This article explains how AI makes mistakes, why those mistakes are not fundamentally different from human mistakes, and how to work with AI the same way you already work with fallible human colleagues. It is written for Australian and New Zealand lawyers who want practical guidance, not performative anxiety.
Let’s start with what you came here for: understanding why AI gets things wrong.
Part 1: Why Legal AI Hallucinates
The Library Analogy
Think about the difference between a librarian and someone who has read thousands of law reports but cannot access the library.
A librarian retrieves exactly what you ask for. They find the specific volume, locate the precise page, and hand you the actual document. If it does not exist, they tell you. There is no guessing.
Someone who has read extensively but cannot access the library works differently. When you ask them about a case, they reconstruct it from memory. They remember patterns—that negligence cases often cite Donoghue v Stevenson, that the Federal Court uses FCA citations, and that the High Court sits above everything else. They give you something that looks right based on everything they know.
Large language models work like that well-read person, not the librarian. They generate text that follows patterns from their training. When you ask for a case citation, the model produces text that looks like a valid citation. Sometimes it matches a real case. Sometimes it does not.
This is not a bug. It is how the technology fundamentally works.
The Technical Version (Without the Jargon)
Large language models predict the most likely next word based on everything that came before. When you type “The leading High Court authority on negligence is,” the model predicts what text most plausibly follows based on patterns in its training data.
This is generation, not retrieval. The model is not searching a database. It is creating text that looks like a correct answer. Sometimes that generated text happens to match reality. Sometimes it does not.
The model learned from text across the internet—legal databases, law firm websites, court decisions, academic articles, blog posts, and everything else. But it did not learn to distinguish between authoritative primary sources and a random blog discussing a case. To the model, it is all just text with patterns.
Where AI Typically Gets It Wrong
Citation accuracy: The model might generate a perfectly formatted citation—say, Smith v Jones [2019] FCA 234—that refers to a case that was never decided. The format is right. The case is fictional.
Jurisdictional confusion: Models trained heavily on American content might default to US legal frameworks or conflate Australian states. They might not recognise that NSW and Victorian approaches differ, or that New Zealand has its own distinct jurisprudence.
Currency: The model does not automatically know that a case has been overruled, a statute amended, or a provision repealed since its training data was compiled. It might confidently cite law that no longer exists.
Misattribution: Even when citing real cases, the model might state propositions the case does not actually support. It has the right authority but the wrong principle.
Confidence without calibration: This is the dangerous one. The model presents uncertain information with exactly the same confident tone as certain information. There is no built-in “I’m not sure about this” signal.
These are real limitations. But before you conclude that AI is uniquely dangerous, consider what happens with humans.
Part 2: The Double Standard—Humans Get It Wrong Too
Every criticism levelled at AI applies equally to human legal professionals. We just accept human error as normal while treating AI error as scandalous.
Humans Hallucinate
Lawyers misremember cases constantly. They recall a principle from a judgment but cite the wrong case. They conflate the ratio of one decision with the facts of another. They remember a statutory provision existing when it has been repealed.
Every senior lawyer has received a research memo citing a case that does not stand for what the memo claims. Every partner has caught a junior citing an outdated version of legislation. Every barrister has seen opposing counsel rely on authority that has been distinguished into irrelevance.
This is so common we have built entire systems around it. Supervision structures. Sign-off processes. Citation checking. Peer review. The legal profession assumes human work product contains errors and builds verification into the workflow.
Why do we suddenly expect AI to be perfect when we have never expected perfection from humans?
Humans Confuse Jurisdictions
New Zealand lawyer cites Australian authority without recognising the different approaches of our courts? How often do practitioners apply English principles without checking whether Australian or New Zealand courts have followed them?
Cross-jurisdictional confusion is endemic in human legal work. It happens in every firm, every week. We just do not write alarming articles about it.
Humans Miss Currency Issues
Lawyers cite overruled cases. They rely on repealed provisions. They miss amending legislation. They overlook that a leading authority has been distinguished or doubted in subsequent decisions.
Professional negligence claims frequently involve lawyers who failed to identify that the law had changed since they last researched the issue. This is not rare. It is a standard category of claim.
Humans Are Overconfident
Partners state legal positions with absolute certainty that turn out to be wrong. Senior lawyers give advice based on recollection without checking. Barristers make submissions based on what they remember from law school rather than current authority.
The confident tone of a senior practitioner is no guarantee of accuracy. We know this. We just extend more grace to human confidence than artificial confidence.
The Verification Standard Has Always Been the Same
Here is the fundamental point: you have never been entitled to accept human work product without verification.
A partner who signs off on a junior’s advice without checking the key authorities is negligent regardless of whether those authorities came from Westlaw, a research memo, a colleague’s recollection, or an AI tool.
The standard has always been: verify before you rely.
AI does not change that standard. It just makes some people suddenly aware of it.
The Missed Opportunity
Here is what the scaremongers miss while they are busy warning you about losing your practising certificate: lawyers who write off AI entirely are handing a competitive advantage to everyone who learns to use it properly.
What AI Actually Offers
Research that used to take hours now takes minutes. Identifying relevant authorities across multiple jurisdictions, summarising lengthy judgments, finding provisions in complex legislative schemes—these tasks consumed enormous amounts of junior lawyer time. AI compresses that dramatically.
Drafting that starts from substance, not a blank page. First drafts of correspondence, submissions, contracts, and advice can be generated in seconds. You still need to refine them—of course you do—but you are refining rather than creating from scratch.
Pattern recognition across vast document sets. Reviewing hundreds of contracts for specific clauses, identifying inconsistencies in discovery, flagging risks in due diligence—AI handles volume that would be economically impossible for humans.
Accessibility for smaller practices. A sole practitioner in Tamworth or Dunedin can now access research capabilities that were previously only available to firms with extensive libraries and dedicated research teams.
More time for the work that actually requires a lawyer. Strategic thinking, client relationships, advocacy, negotiation—the high-value activities that AI cannot replicate get more of your attention when AI handles the groundwork.
This is not theoretical. Firms across Australia and New Zealand are already delivering faster, more cost-effective services using AI tools. The firms that figure this out will outcompete those that do not.
The competitive reality is simple: while you are reading scare pieces about Mr Dayal, your competitors are learning how to use AI properly and taking your clients.
When AI Sees What You Miss
The horror stories always run in one direction: AI gets it wrong, the lawyer fails to check, and consequences follow. But there is another scenario that never makes the headlines.
Recently, a barrister using AI Legal Assistant ran a research query and reviewed the results. One of the AI’s conclusions struck her as wrong. She knew the judgment in question—had read it before—and her instinct said the AI had missed the mark.
But instead of dismissing it, she paused. She pulled up the judgment and read it again.
After a second read, she still was not entirely sure. The AI’s interpretation was not obviously correct, but it was not obviously wrong either. It had a point she had not fully considered.
So she did something sensible: she took the judgment to the judge and said, “This is my reading. This is what the AI suggested. What do you think?”
The judge reviewed it.
The AI was correct.
Had she dismissed the AI’s analysis based on her initial instinct, she would never have included that authority in her materials. A golden opportunity missed because a human assumed she knew better.
This is not to say AI is always right.
It is not.
But the verification process works both ways. Yes, you should check AI output against your own judgment. But you should also be open to AI catching things you missed—because it will. The well-read assistant who has processed more case law than any human could read in a lifetime will occasionally spot what you overlooked.
The lawyers who get the most value from AI are not the ones who treat it as an oracle. They are also not the ones who treat it as a threat to be second-guessed into irrelevance. They are the ones who treat it as a genuine collaborator—capable of being wrong, but also capable of being right when they are wrong.
That is the conversation the scaremongers never have.
Part 3: The Expectation Problem
Much of the anxiety around legal AI stems from distorted expectations about what the technology should do.
AI Is Not a Calculator
When you use a calculator, you expect the answer to be correct every time. Two plus two is always four. The output requires no verification.
People approach AI with the same expectation. They assume that because it is technology, it should be perfectly reliable. When it makes mistakes, they feel betrayed in a way they never feel when a human colleague makes mistakes.
But AI is not a calculator. It is a sophisticated pattern-matching system that generates probabilistic outputs. Expecting perfect accuracy is like expecting a barrister to never misremember a case or a junior lawyer to never make a research error.
We Have Never Had Technology Like This
Previous legal technology was deterministic. Document management systems stored files. Practice management software tracked matters. Legal databases retrieved documents. These systems did exactly what they were programmed to do, every time.
AI is different. It generates novel outputs based on patterns. It reasons (or approximates reasoning) rather than simply retrieving. This is genuinely new, and our mental models have not caught up.
The appropriate comparison is not to a database. It is to a very fast, very well-read, but imperfect research assistant. Would you copy a research assistant’s memo directly into submissions without checking it?
Of course not.
Well, at least I hope you wouldn’t.
You should not do that with AI either.
If AI Were Perfect, You Would Be Redundant
Here is an uncomfortable truth: if AI never made mistakes, there would be no need for lawyers to supervise it. Perfect AI would simply produce correct legal analysis, and clients would use it directly.
The fact that AI requires professional oversight is not a bug. It is what preserves the role of the legal profession. AI handles the volume and velocity; lawyers provide the judgment and accountability.
The limitations of AI are actually aligned with your professional interests. Embrace the oversight role rather than resenting that it is necessary.
Part 4: Beware the LinkedIn Warriors
A word of caution about where you get your information on legal AI.
Scroll through LinkedIn on any given day and you will find dozens of posts from self-appointed AI experts warning lawyers about the dangers of the technology. They share the same handful of horror stories. They speak in ominous tones about regulatory risk. They position themselves as voices of reason in a world gone mad with technological enthusiasm.
Ask yourself: has this person actually built an AI system for legal work?
Have they spent hundreds of hours training lawyers on how to use these tools effectively?
Have they seen what works and what does not in actual legal practice?
Or are they simply repackaging news articles and adding commentary from the sidelines?
There is a world of difference between someone who has coached hundreds of practitioners through the practical realities of AI adoption and someone who read an article about Mr Dayal and decided to post about it.
The loudest voices are often not always the most informed. The people doing the actual work—building systems, training lawyers, solving real problems—are usually too busy to post daily LinkedIn content.
Be sceptical of anyone whose entire contribution to legal AI is telling you what not to do. The valuable guidance comes from people who can tell you what to do because they have actually done it.
Part 5: Working With AI Like You Work With People
Once you stop expecting AI to be a perfect oracle and start treating it like a capable but fallible assistant, the path forward becomes clear.
Apply the Same Standards You Apply to Human Work
If a junior lawyer handed you research, you would:
- Check that key citations actually exist
- Verify that cases are cited for propositions they support
- Confirm the research addresses the right jurisdiction
- Check whether authorities remain current
- Apply your own judgment to the analysis
Do exactly the same with AI output.
The workflow is identical.
The verification steps are identical.
The professional responsibility is identical.
From an insurance perspective, the supervision and verification requirements are likely to be similar whether the work is done by AI or humans.
Invest Appropriately in Verification Based on Stakes
You do not check every sentence a senior associate writes with the same rigour you apply to a first-year lawyer’s work. You calibrate supervision to risk.
Apply the same calibration to AI. Low-stakes internal memos might need lighter verification. Submissions to a court demand rigorous checking. Advice affecting significant client interests requires careful review.
Build AI Into Existing Quality Systems
Your firm already has processes for ensuring work quality: supervision structures, sign-off requirements, peer review, precedent systems. AI should slot into these existing frameworks, not bypass them.
AI-assisted research still goes through the same review process as human research. AI-drafted documents still require the same approval workflows. The quality control mechanisms you have built over years remain valid; they just apply to a new source of work product.
Choose Tools Designed for Legal Work
Not all AI is equal. The lawyers who got in trouble were typically using general-purpose chatbots like ChatGPT—tools designed for conversation, not legal research.
Legal-specific AI tools should provide:
- Source citations you can verify against authoritative databases like AustLII, NZLII, Jade, or Westlaw
- Retrieval from verified legal sources rather than generating from general training data
- Jurisdiction controls that limit results to Australian or New Zealand law
- Confidence indicators that flag uncertainty
- Audit trails showing what was retrieved and how conclusions were reached
When evaluating any tool, ask: does this help me verify the output, or does it expect me to trust blindly?
Good tools make verification easy.
Bad tools obscure their reasoning.
Part 6: Practical Tips for Better Results
Even with appropriate expectations and good tools, how you interact with AI significantly affects output quality.
Be Jurisdiction-Specific From the Start
Weak query: “What are the requirements for a valid will?” There is too much ambiguity in this question. Which jurisdiction? How many parties? What types of parties? Etc, etc.
Stronger query: “What are the formal requirements for a valid will under the Succession Act 2006 (NSW) for a testator with capacity?”
Specifying jurisdiction, legislation, and relevant circumstances eliminates ambiguity the AI would otherwise guess at.
Provide Legal Context, Not Just Facts
Weak query: “My client was fired after complaining about safety.”
Strong query: “Advise on potential general protections claims under Part 3-1 of the Fair Work Act 2009 (Cth) where an employee was terminated following workplace safety complaints. Focus on adverse action and the reverse onus provisions.”
The AI does not know which legal framework you want unless you specify it.
Break Complex Research Into Stages
Weak approach: “Advise on liability for the accident.”
Strong approach:
- “What are the elements of negligence under Australian law?”
- “What is the leading authority on the duty of care owed by occupiers to lawful visitors?”
- “How have Australian courts applied the standard of care where the defendant knew of a risk but assessed it as unlikely?”
Staged queries produce more precise, verifiable outputs. You can check each stage before proceeding.
Use Negative Constraints
Tell the AI what you do not want:
- “Do not rely on US or UK authorities unless expressly adopted in Australian jurisprudence.”
- “Exclude superseded legislation.”
- “Focus on binding authority from the High Court or relevant intermediate appellate courts.”
Negative constraints rule out common failure modes.
State Your Purpose
“I am drafting submissions for an interlocutory injunction in the Federal Court” produces different output than “I am advising a client on litigation prospects” for the same legal question. Context shapes relevance.
Verify Every Citation You Rely On
This remains non-negotiable. A thirty-second check on citations provided confirms whether a case exists and whether the cited proposition appears in the judgment. In the situation where there is no citation provided but it sounds like something you would want to use, a quick search on Austlii or NZlii is always worth the time. Either you validate the citation and use it to bolster your case or you remove it and have confidence in the submission. This is basic professional practice; I should not need to say regardless of whether the citation came from AI, a colleague, or your own memory.
The Bottom Line
The Victorian Solicitor did not lose his ability to run a practice because AI is dangerous. He lost it because he did not do his job. He submitted material to a court without verifying it. That has always been misconduct, regardless of where the material came from.
The lawyers in every AI horror story made the same mistake: they treated AI output as a finished work product rather than a starting point requiring professional review. They would have faced identical consequences if they had submitted unverified work from any other source.
AI is a tool. Like every tool, it can be used well or poorly. Used well, it dramatically expands what you can accomplish. Used poorly—which means used without the verification you would apply to any work product—it creates liability.
The choice is not between safe avoidance and dangerous adoption. The choice is between thoughtful integration and competitive irrelevance.
The lawyers who thrive will be those who treat AI like they treat capable but fallible colleagues: leveraging their strengths, checking their work, and taking responsibility for the final product.
That is not a new standard. It is the standard that has always applied.
AI just makes it obvious.
Post a comment
You must be logged in to post a comment.
Related Posts
My Submission to the NSW Supreme Court on AI in Legal Practice: Why Your Voice Matters Too
My Submission to the NSW Supreme Court on AI in Legal Practice: Why Your Voice…
Why the ACT Bar Association’s AI Training Partnership Marks a Turning Point for the Legal Profession
Why the ACT Bar Association’s AI Training Partnership Marks a Turning Point for the Legal…
AI Legal Assistant Announces Partnership & Integration with Actionstep
AI Legal Assistant Announces Partnership & Integration with Actionstep Delivering bi-directional integration that cuts document…
Lawyers: The Real Risk Of Using American Legal AI Tools
Lawyers: The Real Risk Of Using American Legal AI Tools Why Australian Data Residency Doesn’t…
