🔍 Read the full analysis: A Second AI Look At Claude’s Model Edit Caught Bugs Tests Missed on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A report published by ThorstenMeyerAI.com says OpenAI’s GPT-6 Astra reviewed a Claude Opus 5.5-led experiment editing the weights of the local model Ornith 1.5. The reviewer found reproducibility and command-line faults that passed unit tests, and measured changes to model weights even when the edit strength was set to zero. The review did not independently repeat the full experiment or establish that the edit improved legal performance.
A review by OpenAI’s GPT-6 Astra found reproducibility and command-line bugs in a Claude Opus 5.5-led experiment that edited the weights of local model Ornith 1.5. According to a report published by ThorstenMeyerAI.com and updated November 9, the faults included a run-resume process that could reuse old results and a weight-format conversion that altered model values even when the edit strength was zero. The findings matter because passing tests did not catch several problems that could have weakened the experiment’s measurements.
The Claude session handled the experiment: it planned a directional weight edit, built the test rig, ran benchmark tasks and organised blind grading by three Claude agents. GPT-6 Astra, operating in Codex, was assigned a separate role: review the implementation without changing the main rig. The source says the reviewer did not inspect the hidden benchmark prompts, topics, answers or reference solutions, and did not produce a competing edited model.
In its first review on November 4, GPT-6 Astra reported four reproducible faults. Resuming an experiment could treat an earlier answer as complete without checking whether the model or sampling settings had changed. A mistyped task ID could result in a run that ended successfully but evaluated no questions. A comma conflict made the real command-line invocation fail even though three parser unit tests passed. And converting weights back to a compact format changed values even with no intended edit.
The report gives a measurement from a real matrix of 1,048,576 values: at zero edit strength, the conversion changed 105,315 reconstructed values, or 0.84% of the matrix, according to the initial test. The source says fixes included rejecting unknown task IDs, changing the sampling syntax and storing edited tensors in higher-precision bf16 rather than re-quantising them. These figures describe weight conversion, not legal quality or a general model error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Found More
The episode shows why a green test suite alone cannot establish that an AI experiment ran as intended. Unit tests can pass while a real entry point fails, and a run can appear complete while relying on answers from a different configuration. In this case, those issues could have affected whether benchmark results were new, correctly targeted and reproducible.
The review also draws attention to a less visible source of error: changing a model’s weight format can itself change the weights. If a study attributes a performance difference to a deliberate edit, conversion effects need to be measured separately. The source report says a later reviewer round found that all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. Those checks concern traceability and evidence about the run; they are not a score of model quality or an error rate.
There are limits to what the result establishes. The review was performed by another AI system, not an independent human audit or a replication of every experiment. It supports the value of checking real commands and data provenance, but it does not by itself show that the proposed edit improved the model’s legal reasoning.
Products Worth Considering
DJI Mini 4 Pro Fly More Combo Drone with DJI RC 2
DJI Mini 3 Drone with DJI RC and 3-Axis Mechanical Gimbal
Sonos Arc Ultra Soundbar with Dolby Atmos, Black
As an affiliate, we earn on qualifying purchases.
The Ornith 1.5 Editing Test
The review concerns a separate experiment described in an earlier ThorstenMeyerAI.com article. In that work, a Claude Opus 5.5 session attempted to improve Ornith 1.5 at law through a directional ablation inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory. The method sought a “loop direction” in the model and weakened it directly in the weights, without training.
The updated source reports results across 10 published Lexassay benchmark tasks: the edited model answered all 10, compared with four for the original, while the legal score remained zero. The task runs are public on Lexassay, the report says. Answering a task is not the same as answering it correctly, and the unchanged score does not establish a legal-performance gain.
The two AI sessions began work on November 4. Once the researchers realised they were working on the same project, they divided responsibilities: Claude retained control of the experiment and grading, while GPT-6 Astra reviewed the implementation. The source describes the review as an AI experiment audit, not legal advice.
Limits of the Review Evidence
The report says GPT-6 Astra did not rerun every test during its final documentation step and was not a fourth blind grader. It also measured weight conversion on a CPU rather than through the production GPU path. The findings therefore identify specific faults and measurements, but do not amount to a complete independent replication of the experiment.
The source’s updated description gives a later conversion measurement of 0.51% of one matrix, with 915,360 of its 1,048,576 values changed. That differs from the initial zero-strength test’s reported 0.84% and 105,315 changed values. The report characterises these as measurements of conversion, but the supplied material does not explain the difference between the two figures in enough detail to reconcile them. Neither figure measures legal quality.
It is also unclear from the available account how much the identified faults affected the originally reported benchmark results, beyond the specific reproductions described. The reported benchmark counts and legal score should not be read as proof that the edit caused an improvement in legal capability.
Public Runs and Follow-Up Checks
The report says the experiment’s 10 published task runs are available on Lexassay, alongside the weight-edited model, an edit control and model-card sampling. Those materials give readers a way to inspect the reported results and compare the edit with the control, although the source does not describe a forthcoming independent replication.
The account says the Claude session made the fixes identified in the first review, each paired with a test intended to fail on the old code. Further clarity would come from documenting how the two weight-conversion measurements differ, showing the effect of the fixes on the original runs, and repeating relevant checks on the production GPU path. Until then, the central finding is about software reliability and measurement provenance, not a confirmed improvement in legal performance.
Key Questions
What did the second AI reviewer do?
GPT-6 Astra, running in Codex, reviewed the implementation of a model-editing experiment led by Claude Opus 5.5. The source says it reproduced faults and checked measurements without changing the main test rig or inspecting hidden benchmark tasks.
What bugs did the review find?
The first review reported that runs could reuse results from another configuration, unknown task IDs could produce empty runs marked successful, and the real command line could fail despite passing parser tests. It also found that converting weights changed values at zero edit strength.
Did the model show better legal performance?
The source reports that the edited model answered 10 published tasks, compared with four for the original, while the legal score remained zero. Those figures do not establish a legal-performance improvement; answering a task does not mean the answer was correct.
Were all the reviewer’s findings fully verified?
The source says the reviewer did not rerun every test during its final documentation step and did not measure on the production GPU path. Its findings concern specific reproduced problems and checks, not a complete independent replication of the experiment.
Where can readers inspect the experiment?
The report says the 10 published Lexassay runs are public, along with the weight-edited model, an edit control and model-card sampling. The source does not say that hidden benchmark tasks were made public.
Source: ThorstenMeyerAI.com
