🔍 Read the full analysis: Z.ai’s Legal AI: What To Make Of Its Benchmark Results And Ranking on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria pass measure in Vals AI’s Harvey Legal Agent Benchmark, ranking seventh among 81 models in the archived leaderboard. It passed 92.5% of individual rubric criteria, but the task score counts a task only when every criterion passes. Z.ai’s claimed 12.5-percentage-point improvement over the base model matches the listed results; its claim of state-of-the-art open-source performance is less clearly substantiated by the materials provided.
Z.ai’s GLM-5.3-FlashX-Legal scored a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark, placing seventh among 81 models on the archived leaderboard. The result measures how often the system met every requirement on a task, not the percentage of legal work it completed overall; Vals also reported a 92.5% individual-criteria pass rate.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its documentation says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of simulated legal-workflow environments. The company recommends using it as an agent connected to services supplying laws, cases and company information.
Vals AI’s archived Version 1 leaderboard lists the legal model at 19.17%, or about one fully passed task in five, and at $0.15 per test. The GLM-5.3-Flash base model scored 6.67% on the same table. That makes the difference 12.50 percentage points, consistent with Z.ai’s stated improvement of 12.5 points. The comparison is within that leaderboard run; it does not establish performance across other tests or real legal matters.
The benchmark’s all-pass score is deliberately strict. Harvey says a task is complete only if every criterion in its rubric passes. Vals reports that GLM Legal met 92.5% of individual criteria, but a single missed requirement can prevent an entire task from counting as passed. The two figures describe different measures and should not be treated as contradictory.
Why the Two Scores Differ
The headline task score gives readers a measure of end-to-end reliability under a demanding rubric, while the criteria score shows how often individual requirements were met. A legal work product can be largely correct yet fail the benchmark’s all-pass threshold because of a missed deadline, citation, recommendation or formatting requirement. That distinction matters when evaluating a system intended to support work where one overlooked detail can change the usefulness of a draft.
The results are relevant to law firms and legal departments considering AI tools, but they do not establish that the model is ready to handle client work without supervision. Z.ai’s documentation frames outputs as initial drafts for legal professionals. Buyers still need to assess citation accuracy, review obligations, data handling and the cost of checking outputs. Vals’ listed test cost is not the same as the model’s token pricing or the total cost of a production workflow.
The ranking also offers a comparative signal, not a blanket endorsement. Seventh place among 81 models indicates where this submission stood on this version of this benchmark. The result does not show that it outperforms every open-source alternative in every legal task, and the supplied material does not define the comparison group behind Z.ai’s broader “state-of-the-art” wording.
Products Worth Considering
FLEXISPOT 60-Inch Executive Standing Desk with Drawers and CPU Cabinet
FLEXISPOT EN1 Electric Standing Desk, 60 x 24 in, Black
FLEXISPOT 40-Inch Height-Adjustable Standing Desk Converter with Keyboard Tray, Black
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 legal practice areas. Each task combines an instruction and client-matter materials with a work product to be reviewed. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria covering details such as facts, conclusions, citations, deadlines, dollar amounts and recommendations.
For its public leaderboard, Vals AI uses a private holdout set of 120 tasks that it says mirrors the benchmark’s public mix. Internet access is disabled during testing. Two language-model judges grade submissions, and the reported task score is the average of their pass rates. Vals’ archived leaderboard lists GLM Legal at 19.17%, while Z.ai rounds that figure to 19.2% in its announcement.
Harvey’s own May 26 baseline found a highest all-pass score of 7.1% among the frontier models in that particular evaluation and said the models collectively completed fewer than 10% of tasks end to end. That was a separate run and should not be read as a direct head-to-head comparison with Vals’ later results. It does, however, help explain why a score near one in five can rank highly even while leaving substantial room for missed requirements.
“An all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.”
— Z.ai, in its model documentation
Limits of the Published Evidence
The benchmark does not settle real-world reliability. It measures performance on a fixed set of tasks under a specified grading process. The supplied materials do not establish how results transfer to a particular firm’s matters, jurisdictions, document collections or review procedures. Nor do they establish that the model can safely produce final legal advice without professional checking.
Z.ai calls the result state of the art among open-source models in the evaluation, but the supplied announcement does not identify its comparison set, and the available materials do not provide published weights for the legal model. That leaves the meaning of “open-source” and the basis for that ranking claim unclear. The leaderboard’s seventh-place overall ranking is specific and documented; the broader claim needs more detail to assess.
The source article also describes a four-task spot check in which the author says GLM Legal cited rulings they could not find. The author explicitly says this was not a benchmark. That small check cannot establish the model’s general citation error rate, and the underlying examples are not included in the supplied material. Citation verification therefore remains a question for independent, reproducible testing. The documentation and blog information provided also do not specify who processes client data, under what terms, or what safeguards apply to particular deployments.
What Buyers Should Verify
The immediate next step for evaluators is to inspect the benchmark methodology and test the model on representative, independently reviewed legal tasks. Firms considering deployment will need to verify every citation and material factual assertion, document who is responsible for reviewing drafts, and establish how client information is handled before connecting the system to internal files or external legal databases.
Z.ai’s AutoClaw blog post, dated October 13, 2026, said the model beta was open by application. The company’s documentation page has no date, while Vals lists the release as October 12. The provided material does not state when broader access will begin or describe any later leaderboard update. Further evidence on model access, independent testing, deployment terms and performance on new benchmark versions will clarify how much weight buyers should give the current ranking.
Key Questions
What does GLM Legal’s 19.17% score mean?
It is the share of tasks on Vals AI’s evaluation for which the model met every required rubric criterion, averaged across two AI judges. It is not the share of all individual criteria passed.
How can the model score 92.5% on criteria but 19.17% on tasks?
A task fails the all-pass measure if even one criterion is missed. The model can therefore meet most individual requirements across the test while passing every requirement on a much smaller share of complete tasks.
Did GLM Legal improve on Z.ai’s base model?
On the archived Vals leaderboard, GLM Legal scored 19.17% and GLM-5.3-Flash scored 6.67%. That is a 12.50-percentage-point difference in that evaluation, matching Z.ai’s stated improvement. It does not guarantee the same gap on other tasks or deployments.
Does seventh place prove the model is ready for legal work?
No. The ranking reflects performance on one benchmark version. It does not establish suitability for a particular firm or remove the need for professional review, citation checking and assessment of data-handling terms.
Source: ThorstenMeyerAI.com
