🔍 Read the full analysis: What Happens When OpenAI Trains Agents Inside Software? Check Ironclad on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI’s October 6 post describes training GPT-6 Astra in hosted copies of contract-management platform Ironclad, using 11 legal, commercial and procurement tasks. Astra met an average 55% of task rubric criteria, while estimated completion times were simulated—not measured customer savings—and the results do not establish readiness for unsupervised contract work.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra on legal, commercial and procurement workflows inside hosted copies of Ironclad’s contract-management software. The reported result—an average 55% of rubric criteria met across 11 tasks—shows progress on specialized software use, but does not establish that agents can safely handle contract work without human review.
The project tested whether models could follow business rules while carrying out multi-step work in specialized software and check whether the result met the task’s requirements. Ironclad staff and OpenAI employees who use the product selected 11 tasks, including setting up nondisclosure agreements, creating procurement approval processes and changing a reusable clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes per task.
Tasks were graded against rubrics with 8 to 50 criteria, depending on complexity. OpenAI reported that GPT-6 Astra met an average 55.0% of criteria, compared with 41.6% for GPT-5.6 Sol at the high setting. An internal model used during Astra’s development scored 63.7%. Astra met about 94% of the criteria on one showcase task, but that example does not describe its overall performance.
OpenAI also reported estimated times per attempt of 19.2 minutes for Astra and 37.0 minutes for GPT-5.6 Sol. The company’s footnote says those figures are simulated estimates based on assumed processing and generation speeds, not observed customer time savings. They apply to the 11 research tasks, not to Ironclad workflows generally.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Rules Matter
The reported score is a share of rubric criteria satisfied, not a share of tasks completed. That distinction matters in contract and procurement work, where a partly correct result can still create a serious process failure. A procurement workflow, for example, may need Finance approval above a spending threshold, Security review for specified requests and Legal review for nonstandard terms. Missing one required condition could route a purchase incorrectly even if the remaining steps were handled well.
OpenAI’s account acknowledges the limitation: losing track of a business rule during a task restricts what a software company can confidently delegate. Ironclad CTO Sunita Verma said agents must preserve “the controls teams rely on.” That makes human checking central to interpreting these results. The test suggests that a model can perform parts of specialized work, but the published figures do not show it completing these workflows reliably enough to remove review.
For software vendors, the project also points to a possible change in how AI capabilities are developed: models can be trained and evaluated within real product workflows rather than only on generic computer-use tasks. OpenAI says it is seeking a small number of software-company partners to contribute difficult tasks, subject-matter expertise, secure testing environments and data suitable for research. If that approach improves agents, vendors may gain more useful automation in their products. It may also make their underlying rules, records and controls more important than the screens customers use to access them.
Products Worth Considering
FLEXISPOT Standing Desk with Monitor Riser & Cable Management
Dell XPS 13 9350 Laptop, 13.4 FHD+ Display, Intel Ultra 7 256V, 16GB DDR5, 1TB SSD
Logitech MX Mechanical Wireless Illuminated Performance Keyboard – Clicky
As an affiliate, we earn on qualifying purchases.
Inside the Ironclad Test
Ironclad is a contract-management software company, not the name of a new agent framework. The source account says OpenAI published two items on October 6; its post about Ironclad drew less attention than a separate publication of 722 mathematics manuscripts. The Ironclad post describes work in which models practised in hosted copies of the company’s product, using tasks based on contract workflows.
OpenAI says it generated synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database and filtered them to remove personal information. The company also states that it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data. According to the post, GPT-6 Astra is the first frontier model trained through this arrangement. These details describe the project as OpenAI presented it; the source material does not provide an independent audit of the data handling or evaluation.
Limits of the Published Results
The results cover 11 selected research tasks, and the source material does not establish how performance would hold up across Ironclad’s wider product, different customer configurations or less predictable real-world work. The 55% figure is an average share of criteria met, while the task-level scores and the specific criteria missed are not provided here. Without those details, readers cannot tell which errors were most common or how often a missed criterion affected a consequential control.
The time estimates are simulated, not measured in customer deployments, so they cannot support a claim of actual productivity gains. It is also unclear from the material how the models were tested against changes in software, unusual contract terms or adversarial inputs, and what safeguards would be in place in a live deployment. OpenAI’s statement about data use is reported by the company; the source material supplies no separate verification. The results therefore do not establish that Astra is available to Ironclad customers or suitable for unsupervised use.
OpenAI Seeks More Software Partners
OpenAI says it is inviting a small number of software companies to work on tasks that current agents cannot reliably complete. It asks prospective partners to bring a concrete example of a failing task, people with deep knowledge of the work, a secure test environment and data that can safely be used for research. The source material does not name additional partners or give a timetable for future projects.
For organizations evaluating AI agents in contract or procurement systems, the immediate next step is to ask vendors for more than a headline score: which requirements were missed, how results were checked, what data was used and what human approval remains necessary. OpenAI has described a research partnership and published initial benchmark figures; wider deployment evidence and independently verifiable customer outcomes are not provided in the material.
Key Questions
What is Ironclad in this report?
Ironclad is a contract-management software company. OpenAI describes training and evaluating a model inside hosted copies of its product, not launching an agent framework called Ironclad.
What does Astra’s 55% score mean?
It is the average share of rubric criteria met across 11 research tasks. It is not the percentage of tasks completed or a measure of customer satisfaction. The source material does not list every criterion Astra missed.
Did the test prove that AI agents save customers time?
No. OpenAI’s reported times—19.2 minutes for Astra and 37.0 minutes for GPT-5.6 Sol—are simulated estimates, not measured customer savings. The source says they apply to the research tasks.
Can companies use Astra to manage contracts without review?
The published results do not establish that. Astra met an average 55% of criteria, and OpenAI’s account says human oversight remains relevant when agents must preserve business rules and controls.
What data did OpenAI say it used?
OpenAI says it built synthetic tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information, and did not use OpenAI customer data, its internal contracts or non-public Ironclad customer data. The source material provides no independent audit of those statements.
Source: ThorstenMeyerAI.com
