🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used a code-based weight edit to reduce looping in the local model Ornith 1.5 35B-A3B. On the 10 published Lexassay tasks, the edited model answered all 10, compared with 4 in the matched original setup, but its legal score remained zero and its answers included 55 wrong or invented citations. The result suggests the edit improved completion, not legal knowledge; the test was small and had one run per setting.
A home experiment reported by ThorstenMeyerAI.com found that a targeted edit to the local AI model Ornith 1.5 35B-A3B helped it finish more legal benchmark tasks, but did not improve its legal score. On the 10 published Lexassay tasks, the edited model answered 10 of 10, up from 4 of 10 in the matched original setup, while its score remained zero.
The experiment ran from Wednesday evening to Thursday morning. A Claude Opus 5.5 session wrote code to run Ornith, detect a recurring internal activation associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The account says the process used no additional training or legal data, and no AI read the weights as text.
The report distinguishes this approach from the weight-space programming method described in an MIT CSAIL post. In that work, GPT-6 Astra reasoned directly about another model’s weights. Here, Claude-generated code measured model behavior and applied a known technique called directional ablation. OpenAI’s GPT-6 Astra, running in Codex, separately reviewed the code, figures and claims; the author says it identified errors that were corrected.
On the edited model’s 10 answers, graders found 7 of 50 checked criteria met, compared with 3 before editing. The answers also contained 55 wrong or invented citations, according to the report. A separate run using sampling settings recommended by the model card, without a weight edit, answered 9 of 10 tasks. That result complicates any claim that the edit alone explains the improved completion rate.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
The Edit Helped the Model Finish
The experiment draws a distinction between a model that completes a response and one that answers accurately. Reducing repetitive output can make a system easier to test or use, but the reported legal results show that finishing a task does not establish that its reasoning or citations are reliable.
For readers evaluating AI tools, the result is a small example of a possible diagnostic intervention: if a model’s failure comes from getting stuck, changing an internal direction may help it produce an answer. It is not evidence that the same method improves legal knowledge, works across models, or is ready for practical legal use. The report itself says the scores are historical experiment results, not a guarantee of future performance, and the article is not legal advice.
Products Worth Considering
Samsung 32-inch Odyssey OLED G8 (G80SH) 4K Gaming Monitor
Dell UltraSharp U2424H 24-Inch FHD Monitor
Samsung 32 Odyssey QD-OLED G8 Gaming Monitor – 4K, 240Hz
As an affiliate, we earn on qualifying purchases.
A Test Inspired by MIT Research
The experiment followed an MIT CSAIL report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, which described an AI directly editing another network’s weights. In the MIT team’s Qwen example, the model’s looping stopped and its reported maths score rose from 7.9% to 48.1%. The home test was related in aim, but did not reproduce that method.
Lexassay tests models on legal questions covering German, U.S. and EU law. Its benchmark has 15 questions; 10 are published and five remain hidden. The report says Ornith, a mixture-of-experts model intended for coding and agent work, scored 1 out of 100 on the benchmark. On the published tasks, two of its answers previously ran into a limit of 62,000 output tokens, with repetitive text in the cut-off transcripts.
The author says other candidate models were unsuitable for this particular test: some failed through lack of knowledge rather than looping, while DeepSeek V4 Flash was too large for the machine and ran under a different token budget. The reported experiment therefore focused on one model and one targeted failure mode.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, the report’s author, summarizing the result
Small Test, Uneven Comparisons
The results are limited to 10 published tasks, with one run per setting. The report also identifies a small development set, a loop detector that changed during the experiment, and three graders from the same model family as limitations. Those factors make it difficult to determine how stable the results would be across repeated runs or independent evaluations.
The comparison is not fully consistent across setups: the unchanged model answered 8 of 10 in its published run, but answered 4 of 10 in the other setup used for the matched comparison. A separate model-card sampling run reached 9 of 10 without editing. The article does not establish how much of the edited run’s completion improvement came from the weight change rather than the setup or sampling differences.
It is also not clear whether the edit would generalize to other models, legal questions or hidden benchmark tasks. The report gives no evidence that it improves legal accuracy beyond the small increase in checked criteria, and its citation errors remain a serious limitation.
Public Runs Offer Further Checks
The report says the runs and answers are public on Lexassay.com, along with grading and citation checks, and that its figures refer to the 10 published tasks. Readers can inspect the original run, the model-card sampling run, the edited model and an edit control, as well as the experiment’s method page.
The next useful checks would be repeated runs under the same settings, independent grading, and evaluation on the five hidden tasks or other held-out legal questions. Until those results are available, the finding is best treated as a limited demonstration that a targeted edit may reduce looping in one local model—not as evidence of improved legal competence.
Key Questions
What did the experiment change?
Claude Opus 5.5 generated code that identified a direction associated with looping and removed it at half strength from the output weights of 27 of Ornith’s 40 layers. The report describes this as directional ablation, not direct AI reasoning over the weights.
Did the edited model become better at law?
Not by the benchmark’s overall legal score: it remained at zero. Graders marked 7 of 50 checked criteria as met, up from 3, while the edited answers contained 55 wrong or invented citations.
How many benchmark tasks did the edited model answer?
It answered 10 of the 10 published Lexassay tasks, compared with 4 of 10 in the matched original setup. The report also records a separate unedited run that answered 9 of 10 using model-card sampling settings.
Was this the same method used in the MIT research?
No. The MIT report described an AI reasoning directly over another model’s weights. This home experiment used code to run the model, measure an internal direction linked to looping, and apply a known weight-editing technique.
Can the result be generalized to other models?
The report does not establish that. It covers one model, one experiment and one run per setting, with a small task set and other acknowledged testing limits.
Source: ThorstenMeyerAI.com
