🔍 Read the full analysis: Our AI Legal Test: 15 Questions Across 42 Setups on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer AI reports that its Lexassay test assessed 42 AI setups on 15 legal questions spanning German, US and EU law. The top score was 79 out of 100, while the legal-focused GLM-5.3 FlashX Legal scored 8; across published runs, the test found 804 citations that were invented or did not support the answers attributed to them.
Thorsten Meyer AI reported on November 4, 2026 that its legal AI test, Lexassay, evaluated 42 model setups against 15 questions on German, US and EU law. In the published results, Claude Opus 5.5 led with 79 out of 100, while the law-focused GLM-5.3 FlashX Legal scored 8; the test also identified 804 invented or misattributed citations across its published runs.
Lexassay’s 15 questions include 10 public tasks and five hidden tasks. They address deadline calculations, recent legal changes, court decisions and questions containing false premises. Meyer says the models answered without internet access or tools, aiming to measure what a model could produce without external research. Different reasoning-effort settings were counted separately, which accounts for multiple setups per model.
According to the results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test says the leader fully met every criterion on eight of the 10 public tasks. GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, scored 8 and fully solved none of the 15 tasks. It provided no answer on two tasks after using its stated reasoning budget, according to Meyer.
The report says citation checks across all published runs found 2,644 citations that were real and 804 that were invented or did not support the claim made. Lexassay deducts points for those errors. Meyer also says the evaluation is a spot check, with one answer per model and task, and that small score differences may reflect noise rather than stable performance.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The results highlight a practical risk for anyone using a chatbot to understand a legal deadline or estimate a claim: a fluent answer can still be wrong, and a citation that looks authoritative may not support it. In legal matters, errors can affect whether a case proceeds, what a person expects to recover, or whether they meet a filing requirement.
Meyer illustrates the point with an example involving two German civil judgments served on April 27, 2027. He says the one-month appeal period would fall on a public holiday in Bavaria but not Berlin, producing different appeal deadlines: May 28 in Munich and May 27 in Berlin. The example is part of the test’s subject matter, not a substitute for advice on an actual case. The report recommends checking laws and decisions in their original sources and consulting a lawyer when a deadline, money or an authority is involved.
The finding is also relevant to companies selling legal AI. A product marketed for professional workflows may not outperform general models on every task, while a higher test score does not by itself establish that a system is safe or suitable for legal work. The results measure performance under Lexassay’s particular conditions, not the full range of real-world legal practice.
Products Worth Considering
FLEXISPOT 36-Inch Standing Desk Converter, Black
FLEXISPOT 60 x 30-Inch Bamboo Electric Standing Desk with White Frame
Samsung Odyssey G4 25-inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
The test compares answers with a gold answer containing three to six weighted criteria. Meyer says several Claude agents graded answers blind, and that each citation was checked against official databases. Citations were categorized as real, misattributed, invented or not findable. A misattributed citation refers to a real law or ruling that does not say what the model claims.
Lexassay also deducts five percent of a task’s points when an answer lacks a “not legal advice” note or a pointer to a lawyer. The test includes hidden questions to limit the chance that future models can reproduce answers found online; only the hidden-set score is published. Meyer says the full method and public answers, gold answers and citation checks are available for scrutiny.
The report discloses that Thorsten Meyer AI runs Lexassay and that Claude agents grade the responses. Meyer notes that Claude Opus 5.5 ranks first, presenting this as a conflict of interest. The results should be read with that disclosure and the test’s limitations in mind. They are not a direct comparison of AI products that can search the web or use legal databases, because the evaluated models had no internet access or tools.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as described in its developer documentation and quoted by Thorsten Meyer AI
Limits of the Published Scores
The reported scores do not establish how the models would perform with web search, legal databases or professional review, since Lexassay tested answers without internet access or tools. The source describes one answer per model and task, making the results a spot check rather than a repeated evaluation across many runs. It does not provide enough evidence to treat small score gaps as reliable differences in capability.
The test’s grading also reflects its own gold answers, weights and citation penalties. The report does not establish how performance would generalize to other legal systems, practice areas or the varied facts of a real client matter. The source discloses the publisher’s ownership of Lexassay and use of Claude agents for grading, but the extent to which those factors affect the rankings is not established by the results presented.
The article excerpt gives detailed results for GLM-5.3 FlashX Legal and selected leading models, but does not supply a complete account of all 42 setups. It also does not provide independent replication of the findings. The published materials may allow readers to inspect answers and citation checks, but no external validation is described in the source.
Review the Answers and Method
Readers can consult Lexassay’s method page and published answers, gold answers and citation checks, according to Meyer. Those materials allow people to examine how individual responses were graded and which references were judged inaccurate. The source does not announce a date for another test or specify a planned follow-up evaluation.
For legal questions, the report’s immediate recommendation is to use AI only as a starting point, verify every cited law or ruling against the original source, and seek a lawyer’s help when a deadline, financial claim or communication with an authority is involved. The Lexassay findings are a test of model outputs, not legal advice and not a guarantee of how any system will perform in a particular case.
Key Questions
What did the Lexassay test evaluate?
It tested 42 AI setups on 15 questions about German, US and EU law, including deadlines, legal changes, court decisions and false-premise questions. Models answered without internet access or tools.
Which model scored highest?
According to Thorsten Meyer AI, Claude Opus 5.5 scored 79 out of 100, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher discloses that Claude agents graded the answers and that Claude Opus 5.5 ranked first.
How did the law-focused GLM model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and, according to the report, fully solved none of the 15 tasks. It gave no answer on two tasks after using its reasoning budget.
What does the report mean by citation errors?
Lexassay counted a citation as misattributed when a real law or ruling did not support the model’s claim, and as invented when the cited item did not exist. Across published runs, the test reported 804 citations in those two categories.
Can I use an AI chatbot to calculate a legal deadline?
The report advises against relying on a chatbot for deadlines. It recommends checking the relevant law and decision in their original sources and asking a lawyer when a deadline or other legal consequence is involved. This article and the test are not legal advice.
Source: ThorstenMeyerAI.com
