This post was originally published on this site.
When a man sitting at a New Jersey bar noticed another patron left his phone behind, he grabbed it and tried to send money to himself through Zelle, Venmo and CashApp. The money never arrived.
The man consulted with ChatGPT, according to Mary Warner, a criminal defense attorney in New Jersey. It told him that since the money had not been transferred, he would not face criminal prosecution.
ChatGPT was wrong.
“AI told him there was no victim. However, when he came to my office, I told him that there was a victim,” Warner told Straight Arrow. “There was a warrant out for his arrest.”
The episode illustrates a new issue that criminal defense attorneys face: Some clients are turning to artificial intelligence for legal advice before they even speak with an attorney.
“This is an issue that criminal defense attorneys are experiencing in our profession,” Warner said. “Clients often come to us having already done research through AI. But oftentimes it’s misleading as to what the strategy should be. People often want the law to be logical, but oftentimes the law isn’t necessarily logical or fair, or the public policy is such that at first blush it does not seem logical.”
Some people aren’t working with attorneys at all. There has been an uptick in people choosing to represent themselves pro se — without a lawyer — because they are leaning on chatbots to help their defense.
“The issue is that clients do research on ChatGPT or some other AI platform and they don’t know how to frame the question,” Warner added. “So AI, which is a great tool, follows your lead; if you’re pointing it in the wrong direction, you’re getting the wrong information.”
Lawyers themselves increasingly use AI, with 69% of legal professionals incorporating AI into their practice, according to a March 2026 legal analytics report from 8am. Still, 60% of law firms lack guidance on how to properly use AI. Some legal professionals have faced repercussions, like monetary fines and suspensions, after submitting documents that used AI but contained errors because the professionals did not check the work.
Now clients are leaning on chatbots too. But not only can AI get it wrong, lawyers said, there are certain things AI cannot offer that a human lawyer can.
The ‘wrong questions’
One of the central problems, Warner said, is that a chatbot can only respond to the facts and questions a user provides.
“Clients are not getting the right answer with AI because they don’t know how to ask the right question,” Warner said. “It makes the interaction with the client harder, rather than helping.”
Patrick Barone, a Michigan criminal defense attorney and member of the National Association of Criminal Defense Lawyers AI task force, said the client’s “research” on AI tends to be “useless and frustrating because rebutting it with the reality as we know it from decades of practice is so time-consuming,” Barone told Straight Arrow.
Barone has been handling DUI cases for 35 years.
“This allows me to use AI intelligently. I can ask the right questions,” Barone told Straight Arrow. “[A]nd more importantly, I have the base knowledge to determine the validity of the output. A lawyer must only use AI within their domain of expertise. I wouldn’t use AI to draft an estate plan. Why? Because I can’t validate the output.”
A person may enter a question based on their incomplete or inexperienced understanding of a case. AI could then produce an answer based on a misguided premise.
“All of a sudden, people are coming out of the woodwork. They have the right numbers, but then they have the completely wrong meaning,” Samuel Jay, a public defender in New Mexico, told Straight Arrow. “They’re making arguments like, ‘You should get me out. You should get my child out, or you should get my husband out of jail or spouse out of jail because they qualify for pretrial conditions of release.’ [But] if you don’t notice the motions filed by the state or the order by the judge … it’ll come out with very different things.”
AI could miss what a lawyer discovers
Another one of Warner’s clients was pulled over while allegedly driving under the influence. Her client consulted a chatbot, Warner said, which told her client there was no way they would not be charged with a DUI.
The advice led the client to believe there was no realistic way to get out of a DUI charge, according to Warner.
“When I reviewed the file and the police reports, I found some inconsistencies that AI had not found, even though I had run the reports through AI,” Warner said.
While a chatbot can identify information or summarize a report, the attorney still has to determine what is relevant and what is germane to the case, while understanding how the legal system works.
When Warner discovered the inconsistencies, it gave her leverage to get her client a reckless driving charge, which is considered less of a penalty than a DUI in New Jersey.
In between the lines
There are many ways a lawyer can help one’s case that AI simply can’t, defense attorneys agree.
Criminal defense is not simply a matter of finding a statute and matching it to a set of facts.
“A good defense attorney can do much that AI can never do,” Michael Holley, a district attorney in Montgomery County, Texas, told Straight Arrow. “A good criminal defense attorney can learn a person’s story and tell it with empathy and intentionality. And a good defense attorney can also advocate for a person in ways that AI can never do.”
Attorneys say that the outcome of a case can rely on factors that are not necessarily written or recorded.
“A good defense attorney will review a case and instinctively know what really matters in this case based on a thousand intangibles that AI will never understand,” Holley said.
A defense attorney, for example, may know that a particular prosecutor prioritizes rehab over prison time.
“Where the law is certainly the majority of it, the other part of it is like poker playing,” Jay, the public defender, said, “[such as] the interpersonal relationships with the court or the district attorney’s office, and knowing what a judge or district attorney wants to prioritize in their argument.”
A lawyer may also understand how to frame their client’s history in a way that gives the judge a fuller picture, which may lessen the terms of a sentence.
“Most crime centers around a very intimate situation — sex, drugs and alcohol — so inherently you have a more personal relationship with a criminal client, rather than a civil client where you’re working through financial and contractual terms,” Warner said. “ChatGPT does not give you the best chance to discuss and explain to your client the gray area, which 90% of cases are in.”
Round out your reading
- From Obama to Trump to Biden: When networks said ‘no’ to presidential speeches.
- ‘The end of American science as we know it.’ White House funding plan sparks alarm.
- AI data centers are popping up everywhere. What exactly do they do?
- The Trump administration is accused of breaking a law built to target the KKK.
- One Republican senator is dead, another is still alive. Both are the center of conspiracy theories.