TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models. The score counts a task as passed only when every rubric criterion is met; the same evaluation reports a 92.5% pass rate across individual criteria. Z.ai’s claim that the model leads open-source models is not independently established by the cited comparison, and a small spot check found citation errors.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, with a 19.17% task pass rate. The result matters because the benchmark scores a task as complete only when every required criterion passes, making the figure a measure of fully satisfied work products rather than average criterion accuracy.
Vals AI’s archived leaderboard lists the model, also called GLM Legal, at 19.17% and reports a test cost of $0.15 per task. The same run records a 92.5% pass rate across individual criteria. That difference reflects the benchmark’s all-or-nothing rule: a missed citation, deadline, or other required element can fail the entire task even when most of its rubric is satisfied.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review, and drafting. The company says it used legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal work. Z.ai recommends connecting the model to legal and company information through agent services. These are vendor descriptions of its development and intended use.
Z.ai says the result is a 12.5 percentage-point improvement over the base model and state-of-the-art performance among open-source models in the evaluation. The leaderboard supports the stated improvement against the listed base-model score. The open-source comparison is less clear: Z.ai does not name the comparison set, and the legal model’s weights were not published in the materials reviewed for the source article. The developer documentation page has no date; Z.ai’s AutoClaw beta announcement is dated October 13, while Vals lists the release as October 12, 2026.
Benchmark brief · Legal AI
How Z.ai’s Legal AI Performs on Benchmarks—and Why It Ranks Near the Top
GLM-5.3-FlashX-Legal placed seventh on Harvey’s Legal Agent Benchmark. Its score measures complete tasks, while the much higher criteria pass rate shows how a single missed requirement can change the result.
Every required rubric criterion must pass for a task to count as complete.
Strong performance across individual criteria does not guarantee a fully passing legal work product.
01 / Reading the result
Two scores, two different views of success
The gap is built into the benchmark’s all-or-nothing task rule. A response can satisfy most of its rubric and still fail the complete assignment if even one required detail is missed.
Pass-rate comparison
Bars share a 0–100% scale. Task completion requires every criterion in that task to pass.
Vals AI lists GLM Legal, also called GLM-5.3-FlashX-Legal, at a test cost of $0.15 per task. That is benchmark testing cost, not the full cost of review or workflow integration.
Z.ai reports a 12.5 percentage-point improvement over its listed base-model score. The leaderboard supports that comparison.
02 / How the benchmark works
A demanding test of complete work products
Harvey’s benchmark tests legal agents on practical assignments with matter materials and requested work products. Its large rubrics reward completeness, so a small omission can determine the task result.
1,250 tasks
Across 24 practice areas, with more than 75,000 expert-written rubric criteria in the public benchmark.
Instruction → matter → work product
Each task pairs an instruction with a client matter’s relevant materials and a requested output.
Many ways to miss
Criteria can check facts, conclusions, citations, severity, recommendations, deadlines, amounts, and formatting.
Instruction and client matter materials.
Agent prepares the requested legal output.
Two model judges grade the submission.
Reported score averages the judges’ rates.
The leaderboard uses a private 120-task holdout intended to mirror the public task mix. Internet access is disabled during evaluation. Vals AI says GPT-5.5 and Claude Sonnet 4.6 judge submissions.
03 / What the evidence says
Useful signal—with clear limits
The leaderboard offers a common comparison point. It cannot, by itself, establish broad legal reliability, comparative error rates, or readiness for unsupervised use.
Vendor description
Z.ai says the model is optimized for document understanding, legal reasoning, research, contract review, and drafting. It describes legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments.
Open-source claim
Z.ai claims state-of-the-art performance among open-source models. The cited materials do not name the comparison set, and the legal model’s weights were not published in the materials reviewed.
Small citation spot check
A four-task review found citations to rulings the reviewer could not find. The sample is too small to estimate error frequency or compare systems reliably.
Operational questions remain
The score does not answer how client information is processed, which rules apply, or what review is required before generated work is shared.
04 / Evaluating for a firm
Questions to answer before adoption
A benchmark result is one measured comparison point. A practical evaluation should test complete outputs and the conditions under which a firm would use them.
Can citations be checked?
Test authorities, quotations, deadlines, and key conclusions against source materials.
Who signs off?
Assess full work products and define the professional review needed before sharing.
How is client data handled?
Determine processing terms, applicable rules, integrations, and broader operating costs.
What did GLM-5.3-FlashX-Legal score?
19.17%. Vals AI’s leaderboard places it seventh among 81 models on Harvey’s benchmark.
Why is that below the criteria rate?
Harvey counts a task only when every required criterion passes. One miss can fail the whole task.
Does this prove it is the best open-source legal AI?
No. The cited materials do not specify Z.ai’s comparison set, and the legal model’s weights were not published in the reviewed materials.
What should firms verify?
Check citations and complete outputs, then assess client-data processing, applicable rules, and human review requirements.
Timing note: Vals lists the release as October 12, 2026; Z.ai’s AutoClaw beta announcement is dated October 13. Its developer documentation page has no date.
Why Full-Task Passes Matter
The score highlights a practical gap between getting most details right and producing a complete legal work product. A high criterion-level pass rate can coexist with a much lower full-task rate when each assignment has many required elements and any one miss counts as failure. That distinction is relevant to law firms evaluating whether an AI assistant can prepare work for review, rather than merely produce plausible text.
The benchmark does not establish that the model is safe to use without human review. In a four-task spot check described by Thorsten Meyer AI, the legal model cited rulings the reviewer could not find. The spot check is small and is not a benchmark result. The same review found no invented rulings in the OpenAI and Claude models it checked, while two Claude models cited a real ruling for the opposite of its holding. Those observations point to citation verification as a concern across systems, but they cannot establish comparative error rates.
Procurement decisions also depend on questions the score does not answer: how client information is handled, which rules govern its processing, and what review is required before generated material is shared. Z.ai lists pricing of $0.37 per million input tokens and $1.25 per million output tokens. Those rates describe model usage costs; they do not measure the expense of verification or integration into a firm’s workflow.
Products Worth Considering
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public benchmark contains 1,250 tasks across 24 practice areas, with more than 75,000 expert-written rubric criteria. Harvey describes each task as an instruction, a client matter containing relevant materials, and a requested work product. Criteria can cover facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts, and formatting.
Harvey says a task passes only when every criterion passes. Its example of a change-of-control assignment has 57 criteria, illustrating why a small number of omissions can reduce the full-task score sharply. The public leaderboard uses a private holdout set that mirrors the public mix; Vals AI says that set contains 120 tasks. Internet access is disabled during the evaluations, according to the benchmark description.
Vals says two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade each submission. Each judge calculates a task pass rate, and the reported score is their average. The leaderboard provides a common evaluation framework, but its results remain tied to this benchmark’s tasks, holdout set, rubric, and judging procedure.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai developer documentation
What the Score Cannot Establish
The result does not show how the model would perform across all legal work or on matters outside Harvey’s benchmark. The leaderboard uses a 120-task private holdout, and the reported score is the average of two model judges’ pass rates. The source material does not provide task-by-task results for GLM Legal or explain how often particular error types occurred.
Z.ai’s “state-of-the-art among open-source models” statement is not fully assessable from the cited materials because the company does not specify the comparison set, and the legal model’s weights were not published in the material reviewed. The four-task citation review is also too small to estimate error frequency. The reviewed Z.ai pages describe outputs as drafts for legal professionals, but none carried a “not legal advice” notice, according to the source article.
Questions for Firm Evaluation
Z.ai’s AutoClaw announcement opened a beta by application, but the source material does not give a timeline for broader availability or further benchmark results. The next useful evidence would include a clearly defined open-source comparison set, additional independent evaluations, and detailed results showing performance by task and error type.
For firms considering the model, evaluation would need to include citation checking and review of complete work products, alongside procurement questions about client-data processing and applicable rules. The benchmark result provides one measured comparison point; it does not answer those operational questions.
Key Questions
What did GLM-5.3-FlashX-Legal score?
Vals AI’s leaderboard lists a 19.17% all-criteria task pass rate on Harvey’s Legal Agent Benchmark and a rank of seventh among 81 models.
Why is the task pass rate lower than the criteria pass rate?
Harvey counts a task as passed only when every required rubric criterion passes. The same run reports a 92.5% individual-criteria pass rate, but one missed criterion can still fail a whole task.
Does the result prove that Z.ai’s model is the best open-source legal AI?
No. Z.ai claims state-of-the-art performance among open-source models, but the cited materials do not specify the comparison set. The legal model’s weights were not published in the materials reviewed for the source article.
What should law firms verify before using it?
Firms would need to check citations and complete outputs, and determine how client data is processed, which rules apply, and what internal review is needed before generated work is shared.
Source: Thorsten Meyer AI
