🔍 Read the full analysis: Same Model, Different Harness: Is CUGA The Right Fit For Your AI Workflow? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In an IBM-reported AppWorld experiment, CUGA completed 73 of 90 tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The results suggest harness configuration can affect performance on this workload, but do not establish an overall ranking or show how the systems perform on production tasks.
IBM reports that its CUGA agent harness completed 73 of 90 tasks in an AppWorld experiment using GLM-5.3-flash, while Hermes completed 42 and Deep Agents completed 27, as detailed in the original analysis. The comparison, which held the model constant across the three setups, suggests that the software coordinating an AI agent can affect its results on a particular multi-step workload; it does not establish which harness is best overall.
IBM says the experiment ran each harness on the same 90 tasks across 30 scenarios, using the Harbor evaluation framework and one attempt per task. CUGA completed 81.1% of tasks, Hermes 46.7% and Deep Agents 30.0%. On a separate scenario measure, a scenario counted as passed only when all three of its variants were completed. CUGA passed 20 of 30 scenarios, compared with eight for Hermes and two for Deep Agents.
The harnesses did not use identical configurations. CUGA used task decomposition, tools focused on the benchmark apps, verification before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP alongside LangChain middleware for tool selection, context summarization, a to-do list, retries and call limits. IBM says it also made two fixes to Deep Agents so malformed model replies would not end a run.
IBM says the experiment setup is publicly available through harbor-eval, its fork of Harbor. AppWorld presents tasks involving nine apps and 457 APIs, including email, payments and file storage. IBM describes the benchmark as a proxy for some enterprise automation capabilities, such as reading API documentation, paging through results and changing information across apps. The benchmark uses consumer apps, however, and does not directly test performance on real enterprise systems.
Why Agent Harnesses Affect Results
The reported difference is relevant to teams comparing AI agents because a model score may not capture how an agent performs once software assigns tasks, selects tools and checks actions. In this test, the same GLM-5.3-flash model was paired with different coordination systems, and completion rates ranged from 30.0% to 81.1%. The result supports evaluating the whole model-and-harness setup against the work it is expected to perform.
It is a useful signal for organizations with workflows that resemble AppWorld’s multi-step API tasks. But the figures are not evidence that CUGA will outperform Hermes or Deep Agents for personal assistance, research or other work. The benchmark also says nothing about policy enforcement: it tested task completion, not whether an agent follows access rules, approval requirements or data boundaries during execution.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Anker SOLIX C1000 Gen 2 Portable Power Station
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Workloads
The comparison is framed as a question of fit, rather than a general contest among agent products. The source describes Hermes, from Nous Research, as aimed at always-on personal assistants, and Deep Agents, from LangChain, as suited to extended work such as research. CUGA, short for ConfigUrable Generalist Agent, is IBM’s open-source harness, presented for enterprise work spanning systems such as customer relationship management and payments.
The source also points to community directories that track nearly 200 agent harnesses, an indication of how many coordination approaches are being developed. It cites a separate directory analysis reporting that GLM-5.2’s SWE-bench Pro result rose from 23% to 52% after a harness change. That is a separate reported comparison, not part of IBM’s AppWorld test, and should not be treated as independent confirmation of the AppWorld results.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
Limits of the AppWorld Evidence
The reported results cover one model, one benchmark and 90 tasks, with one attempt per task. The harness configurations differed, and the supplied material does not explain how much of the performance gap came from particular features or other implementation choices. It is also unclear whether the results would hold with other models, repeated runs or different task sets.
The experiment did not test real enterprise deployments, policy enforcement, cost, latency or performance at larger scale. IBM describes CUGA as having features for applying rules through a harness, but the AppWorld scores do not establish how effective those features are. The source provides no timeline for an independent replication or a broader comparative study.
Testing Agents on Real Workflows
Teams considering these systems can use IBM’s public harbor-eval setup to inspect the reported experiment, then test candidate model-and-harness combinations on workflows representative of their own work. Repeated runs and varied tasks would help show whether a result is consistent rather than tied to a single attempt or task sample.
For enterprise use, evaluations would also need to cover requirements absent from AppWorld, including access controls, approval steps, data boundaries and behavior in the organization’s deployment environment. No next study or publication date has been announced in the supplied material.
Key Questions
What did IBM’s AppWorld experiment find?
IBM reports that CUGA completed 73 of 90 tasks with GLM-5.3-flash. Hermes completed 42 and Deep Agents 27 in the same benchmark comparison.
Does the result prove CUGA is the best agent harness?
No. It is a result from one model and one benchmark, with differently configured harnesses. It does not establish a general ranking across other tasks or uses.
What is an agent harness?
A harness is the software that coordinates an AI agent’s work, including how it breaks down tasks, selects tools and checks actions or results. This experiment suggests those choices can affect task completion.
Did the test evaluate enterprise security or policy enforcement?
No. IBM’s AppWorld comparison measured task completion, not whether systems enforced policies, access controls or approval requirements. Those areas need separate evaluation.
Can organizations inspect or repeat the experiment?
IBM says the setup is publicly available through harbor-eval, its Harbor fork. The supplied material does not give a timeline for independent replication.
Primary source: Hugging Face · via ThorstenMeyerAI.com
