🔍 Read the full analysis: A Qwen JSON Tab Escaping Experiment: 49 Before, 70 After on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An experiment led by OpenAI’s GPT-6 Astra changed one matrix in Qwen3-4B-Instruct-2507 and raised its exact score from 49 to 70 of 120 on a frozen test of copying special characters into JSON. The gain was limited to that task: table-row extraction remained at zero of 30, 50 final answers were still wrong, and the synthetic test has not received an independent review.
An experiment led by OpenAI’s GPT-6 Astra changed one matrix in the open Qwen3-4B-Instruct-2507 model and raised its exact score from 49 to 70 of 120 cases on a frozen test of copying text with tabs and other special characters into JSON. The result is a measured improvement on one narrow, synthetic task—not evidence of a general capability gain—and 50 of 120 answers remained wrong.
The test checked whether the model could copy short source texts into a JSON field without changing any characters. A literal tab must be represented in JSON as the escape sequence \t, then decode back to the original tab. An answer counted as correct only if parsing the JSON recovered the source text exactly; valid JSON with altered or missing characters did not pass.
According to the experiment report, GPT-6 Astra operated as a Codex agent and planned and ran the work. It inspected Qwen’s weights, created and froze the tests, calculated and saved the edit, and reported the result. The intervention was a calculated rank-one update to the output projection of the final MLP layer. It did not use gradient training, and the report says the other 397 stored tensors remained bit-identical to the original.
The final result was 21 more exact answers and no cases made worse across the 120-case test, equivalent to a 17.5-percentage-point increase in the exact-answer rate. Earlier screening gave the unedited model 16 correct answers out of 32; a more explicit prompt also scored 16 of 32, according to the report. The experiment’s source says a separate table-row extraction task remained at 0 of 30.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in JSON Reliability
The finding matters because it offers a specific example of an AI agent modifying a model’s stored weights and then measuring a change in behavior, rather than relying only on new instructions. In software systems that pass data through JSON, mishandling a tab, quote, backslash or line break can corrupt a value. Exact preservation is a practical requirement for some data-transfer and automation tasks.
But the result does not show that the model became more reliable across JSON work generally, or that weight editing is a dependable substitute for prompt design or conventional training. The score covers one targeted task and 120 test cases; half the final cases still failed. The unchanged table-row result also illustrates that the edit did not resolve every weakness tested in the experiment.
The reported 17.5-point gain is the difference between this model’s scores on this experiment’s test, not a measure of performance across users, applications or other models. The cases were synthetic, and the report says a second AI has not yet independently checked the work. Readers should treat it as a limited experiment, not a production benchmark or a guarantee of future performance.
Products Worth Considering
Apple MacBook Pro MPHF3LL/A 14.2-Inch Laptop | M2 Pro 12-Core Chip 16GB Unified Memory 1TB SSD | 19-Core GPU Liquid Retina XDR Display Space
Apple MacBook Neo 13-inch Laptop with A18 Pro Chip, 8GB Memory, 256GB SSD, Liquid Retina Display, and 1080p FaceTime HD Camera – Blush
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
How the Experiment Followed Earlier Tests
The test was the third experiment in a week-long series about editing neural-network weights. The first used Claude to edit a local model called Ornith; according to the source account, it produced more completed legal answers, but the legal score stayed at zero. GPT-6 Astra later reviewed that work and identified reported problems including a resume bug, a broken command-line interface and side effects from weight conversion.
Before the Qwen escaping test, GPT-6 Astra had tried another edit on the same model for extracting fields from synthetic business messages. The frozen final test in that attempt scored 97 of 200 for both versions. The account says both had 47 invalid answers, mostly because raw tab characters appeared where JSON required escaped tabs. That failure supplied the lead for the next experiment: investigate a known tab-escaping weakness rather than search broadly for any possible improvement.
The site owner directed the agent to inspect Qwen and try to improve a weakness on November 5, 2026, according to the report. The source identifies Qwen3-4B-Instruct-2507 as an open model under the Apache-2.0 license. It also says Claude did not participate in this experiment; Claude was used to research and draft the source article and check figures against stored results.
Independent Checks Still Pending
The reported score is clear within the experiment’s stated test, but several limits remain. The source describes the test cases as synthetic and says the work has not yet been reviewed by a second AI, despite the earlier experiment series having included a separate review of the first edit. The reported improvement has not, in the supplied material, been independently reproduced.
The available account does not provide enough detail to judge how well the 120 cases represent real-world JSON use, how the test set was assembled beyond being frozen before the result was measured, or whether the result holds across different prompts, model versions or software environments. It also does not establish that the edit improves JSON handling beyond the specific tab-and-special-character task. Those questions remain open rather than resolved by the score.
Reproduce and Broaden the Test
The next useful step is an independent review of the experiment records, edit procedure and frozen test, followed by reproduction of the result. Further tests could check whether the improvement persists on new examples and whether it extends to varied JSON inputs, without changing the task or test after seeing the outcome.
Until such checks are reported, the confirmed result is limited to the stated comparison: the edited model scored 70 of 120 exact answers, compared with 49 before the edit. The source material does not announce a date for an independent review or another experiment, so the timing and scope of any follow-up are not yet known.
Key Questions
What changed in the Qwen model?
The experiment applied a calculated rank-one update to the output projection of the model’s final MLP layer. The report says the other 397 stored tensors were unchanged.
What does 49 to 70 of 120 mean?
It is the number of final test cases answered with an exact match before and after the edit. The report describes 21 additional correct cases, with none of the 120 cases becoming incorrect.
Did the edit improve all the tested tasks?
No. The report says the table-row extraction task stayed at 0 of 30. The reported gain concerned the targeted JSON escaping test.
Has another reviewer verified the result?
Not yet, according to the source account. It says the experiment has not been checked by a second AI; independent reproduction is also not reported in the supplied material.
Source: ThorstenMeyerAI.com
