TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A test by Thorsten Meyer AI compared 42 AI setups on 15 legal questions covering German, US and EU law. Claude Opus 5.5 scored highest at 79 out of 100, while a model marketed for legal work scored 8; the test’s publisher says citation errors were widespread. The results are a limited spot check, and another benchmark produced a substantially different ranking.
Thorsten Meyer AI has published results from Lexassay, a test comparing 42 AI setups on the same 15 legal questions, reporting a wide spread in scores and errors in legal citations. Claude Opus 5.5 scored 79 out of 100, the highest result in the published comparison, while Z.ai’s GLM-5.3 FlashX Legal scored 8; the findings matter because a wrong legal answer or citation can mislead someone facing a deadline or court process.
The test covered questions on German, US and EU law, including appeal deadlines, changes to laws, court decisions and questions that contained false assumptions. Models answered without internet access or tools, so the test measures what they could produce from the model itself rather than what a search-enabled legal assistant might find. Lexassay included 10 public questions and five hidden questions; the publisher says it releases scores, answer materials and citation checks for public questions, while publishing only scores for hidden tasks.
At the top of the reported results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher says the leading model fully met every criterion on only eight of the 10 public tasks. At the lower end, GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the 15 tasks, according to the test. Mistral Large 4 scored 10; the publisher says it got about half the substance right but lost most of its points over incorrect citations.
Lexassay reports that across its published runs it confirmed 2,644 citations as real and identified 804 that were invented or did not support the claims made about them. Under its scoring rules, each invented or misattributed citation costs a quarter of a task’s points. Answers also lose 5 percent if they omit a legal-advice disclaimer or a suggestion to consult a lawyer. These figures describe Lexassay’s checks and scoring; they are not an independent estimate of citation accuracy across all legal AI use.
42 AI setups faced the same legal questions Scores vary widely
Lexassay tested 42 AI setups on 15 questions about German, US and EU law. Its results are a limited spot check, not a guide to performance in every legal matter.
Scores show a wide gap
The comparison measures answers produced without internet access or tools.
What the test covers
Three legal systems
The 15 questions cover German, US and EU law, including appeal deadlines, court decisions and false assumptions.
Answers checked against references
Each answer is graded against a reference answer and its citations are checked against official databases.
One answer per task
The test does not measure consistency across repeated attempts or performance with search tools and supplied documents.
Rankings depend on the test
A separate legal-agent benchmark supplied documents and produced a different order.
- Mistral Large 4 scored 10.
- Claude Opus 5.5 scored 79.
- GPT-6 Astra scored 72.
- Mistral Large 4 ranked eighth.
- Claude Opus 5.5 ranked 21st.
- GPT-6 Astra ranked 24th.
Treat legal AI answers with care
A plausible answer can still cite a ruling that does not support it, and a deadline can turn on the jurisdiction and facts.
- Check every cited law or ruling against the original source.
- Consult a lawyer when a deadline, money or an authority is involved.
- Do not rely on a chatbot to calculate a live case deadline.
Legal Errors Can Change Outcomes
The results illustrate why legal AI scores need to be read alongside the underlying answers and sources. A system can produce a plausible-sounding response while citing a ruling that does not support it. In deadline calculations, a single day can also affect whether an appeal is accepted. The test gives readers examples of those risks, while its limited design means it cannot establish how often any model will make such errors in routine legal work.
The article’s example concerns two German civil cases served with judgments on Tuesday, April 27, 2027, one in Munich and one in Berlin. According to the source, both have a one-month appeal period, but the end date differs because Corpus Christi is a public holiday in Bavaria: the Munich deadline falls on May 28, while the Berlin deadline remains May 27. The written grounds deadline is June 28 in both cases. This is an illustration in the source material, not legal advice for an individual case.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, Space Black
As an affiliate, we earn on qualifying purchases.
A Test Without Online Research
Lexassay grades each answer against a reference answer, called a gold answer, using three to six weighted criteria. It checks citations against official databases and classifies them as real, misattributed, invented or not findable. The publisher says each model-task pair received one answer, and different effort settings were evaluated separately; a model can therefore appear more than once among the 42 setups.
The publisher describes the exercise as a spot check rather than a strict benchmark. One answer per task cannot show how consistent a model is across repeated attempts, and small score differences may reflect noise. The results also do not test retrieval tools or supplied legal documents. That distinction matters when comparing the findings with Vals AI’s legal-agent benchmark: in that test, where models work with supplied documents, Mistral Large 4 ranked eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th, according to the source.
Product descriptions are separate from test results. Z.ai describes GLM-5.3 FlashX Legal as “optimized for professional legal workflows,” while Mistral calls Large 4 “state-of-the-art among open models” on enterprise work, including law. Those are company claims; the Lexassay scores are the publisher’s results under its own questions and grading rules.
Limits of the Published Scores
The results do not establish how the models perform across the full range of legal work. Lexassay used 15 questions and one answer per model and task, with no internet access or tools. Its publisher also runs the test and says Claude agents graded answers blind. The article discloses that relationship and says the method and public answers, reference answers and citation checks are published for scrutiny. Readers should treat the ranking as one evaluation, not a settled comparison of legal capability.
The source does not provide enough detail here to independently assess how the questions represent real client matters or how results would change with repeated runs, updated models, retrieval tools or human review. It also does not establish that a model’s score predicts the outcome of a particular case. Those limits are especially relevant because another benchmark cited in the article, using supplied documents, ranks several of the same models differently.
Checking Answers Against Legal Sources
Lexassay says its public materials allow readers to inspect the questions, answers and citation checks, while details of hidden tasks are withheld to reduce the chance that later models can learn their answers from the web. The source does not announce a date for a next test or a new evaluation milestone.
For people using AI around a legal matter, the publisher recommends treating chatbot output as a starting point, checking every cited law or ruling in the original source, and consulting a lawyer when a deadline, money or an authority is involved. Do not rely on a chatbot to calculate a legal deadline for a live case; the applicable rules can depend on the jurisdiction and facts.
Key Questions
What did Lexassay test?
It compared 42 AI setups answering the same 15 questions on German, US and EU law, without internet access or tools.
Which setup scored highest?
Claude Opus 5.5 scored 79 out of 100, the highest reported score. The source says it fully met every criterion on eight of the 10 public tasks.
How did the model built for legal work perform?
Z.ai’s GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the 15 tasks, according to Lexassay’s results.
Does the test show which AI is best for legal work?
No. The publisher calls it a spot check, based on one answer per task, and another benchmark cited in the article produces a different ranking. The results do not predict performance in every legal matter.
Should someone use a chatbot to calculate a court deadline?
The source advises checking legal citations against official materials and consulting a lawyer when a deadline is involved. The test is not legal advice, and its example does not determine a reader’s own deadline.
Source: Thorsten Meyer AI
