<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Latent Thoughts]]></title><description><![CDATA[I write practical, hands-on articles about AI engineering, cloud architecture, and software development. Through reproducible experiments, working code, and real-world examples, I explore how to build, evaluate, and operate AI-powered applications. Topics include reasoning models, AI agents, evaluation frameworks, and the trade-offs between accuracy, latency, and cost. My focus is helping developers and architects move beyond demos and make informed engineering decisions.]]></description><link>https://blog.latentthoughts.com</link><image><url>https://cdn.hashnode.com/uploads/logos/6abe4a89fbf68ad459d66120/b90869f0-708e-42a8-9108-c9bb2d905772.png</url><title>Latent Thoughts</title><link>https://blog.latentthoughts.com</link></image><generator>RSS for Node</generator><lastBuildDate>Thu, 12 Nov 2026 17:26:48 GMT</lastBuildDate><atom:link href="https://blog.latentthoughts.com/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Recursive Self-Improvement Meets a Strong Baseline Agent]]></title><description><![CDATA[Can recursive self-improvement make a strong AI agent better? Two retail experiments test how revised instructions affect accuracy, speed, and cost.]]></description><link>https://blog.latentthoughts.com/recursive-self-improvement-meets-a-strong-baseline-agent</link><guid isPermaLink="true">https://blog.latentthoughts.com/recursive-self-improvement-meets-a-strong-baseline-agent</guid><category><![CDATA[#agent]]></category><category><![CDATA[agentic]]></category><category><![CDATA[openai]]></category><category><![CDATA[open ai]]></category><category><![CDATA[Evaluation]]></category><category><![CDATA[evals]]></category><category><![CDATA[llm]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Mon, 09 Nov 2026 16:15:55 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/abb11d50-e56b-4841-84e8-488c6af591ce.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Researchers already use bounded forms of recursive self-improvement (RSI) to train language models. A model can generate training examples or judge its own answers. The model uses that signal to update its weights, and the updated model provides the signal for the next round. <a href="https://proceedings.mlr.press/v235/yuan24d.html">Self-Rewarding Language Models</a>, <a href="https://arxiv.org/abs/2510.18814">A Model Can Help Itself</a></p>
<p>This article tests the thesis that a similar loop can improve an agent’s work while the underlying model stays fixed. In this form of recursive agentic self-improvement, a separate component reviews the agent’s actions and outcomes, then revises its instructions. The next round starts with those instructions and carries forward the evidence from earlier runs, including where a change made performance worse.</p>
<blockquote>
<p>The thesis that a similar [RSI] loop can improve an agent’s work while the underlying model stays fixed.</p>
</blockquote>
<p>The counter-thesis is that a strong baseline agent may leave too little useful headroom for that process to justify itself. The baseline is the agent running with its original system instructions. The first runs show which requirements the baseline meets and which it misses. If it already meets the tested requirements, there may still be room to improve speed or cost. A revision can also add overhead or improve one measure without benefiting the whole workflow. This post examines that tradeoff through a series of experiments built around OpenAI models and APIs, using Codex to help engineer the surrounding system.</p>
<blockquote>
<p>A strong baseline agent may leave too little useful headroom for that process to justify itself.</p>
</blockquote>
<p>We chose retail to see how the same agent handled different kinds of work within one business. A simulated online store gave us tasks with concrete outcomes, from finding a product within a shopper’s budget to reconciling a payout against the store’s records.</p>
<p>We tested twelve different retail tasks and, in a separate study, twelve variations of a return-and-exchange workflow. We measured correctness, task time, and model cost in both.</p>
<h2>Source Code</h2>
<p>All open-source code used to produce this post is available on <a href="https://github.com/garystafford/recursive-agent-instruction-improvement">GitHub</a>.</p>
<p><a class="embed-card" href="https://github.com/garystafford/recursive-agent-instruction-improvement">https://github.com/garystafford/recursive-agent-instruction-improvement</a></p>

<h2>Inspecting task outcomes with Promptfoo</h2>
<p>We used <a href="https://www.promptfoo.dev/">Promptfoo</a> to evaluate saved runs in both studies. Its built-in assertions and custom JavaScript checks examined the agent’s final reply, recorded actions and approvals, and the resulting store records. For each task, we applied the same checks across instruction versions.</p>
<p>Promptfoo’s trace view shows the sequence of model steps, tool calls, and approvals so we can inspect what happened during a run. The time and cost comparisons use the runner’s task timer and the original model usage records.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/8a12c82f-4f14-41d9-8883-25052c8065fc.png" alt="Promptfoo displays recorded model generations, tool calls, and their timing within a retail task." style="display:block;margin:0 auto" />

<p><em>Figure 1. Promptfoo shows the model steps and tool calls behind a task result.</em></p>
<p>We used the comparison view to see where an instruction revision changed the outcome. Each row aligns the three versions for the same task and pass, and the column headers show how many runs each version passed.</p>
<p>In one payout reconciliation task, the agent correctly explained a discrepancy in its reply. It recorded $160 as the amount settled that day. Promptfoo flagged the mismatch because the correct total was $100.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/5af37674-d36f-41cf-8b00-3b3f33ff2a75.jpg" alt="Promptfoo compares both passes of the twelve retail tasks. Baseline passes 23 of 24 cases, Revision 1 passes 24, and Revision 2 passes 22." style="display:block;margin:0 auto" />

<p><em>Figure 2. Both passes of the twelve retail tasks in Promptfoo, with Baseline, Revision 1, and Revision 2 side by side.</em></p>
<h2>Twelve different retail tasks</h2>
<p>For the baseline, we gave the retail agent the same <strong>system instructions</strong> for every task, along with a workflow-specific request. Those instructions told the agent to check the store’s rules and records, get approval before actions such as placing orders or issuing refunds, and verify the outcome before reporting it.</p>
<h3>Baseline results by task</h3>
<p>Each workflow had one scenario with its own simulated records and success checks. The table shows the baseline averages for each workflow. Each model step is a recorded model generation. Tool calls include business tool calls, scripted approval calls, and local commands. Total tokens include input and output across all model steps, including cached input, rounded to the nearest thousand.</p>
<table>
<thead>
<tr>
<th align="right">#</th>
<th>Workflow and expected outcome</th>
<th align="right">Avg. model steps</th>
<th align="right">Avg. tool calls</th>
<th align="right">Avg. task time</th>
<th align="right">Avg. total tokens</th>
</tr>
</thead>
<tbody><tr>
<td align="right">1</td>
<td><strong>Product selection.</strong> Recommend a waterproof jacket within budget and the delivery deadline.</td>
<td align="right">2</td>
<td align="right">3</td>
<td align="right">20.6 s</td>
<td align="right">45k</td>
</tr>
<tr>
<td align="right">2</td>
<td><strong>Purchase and checkout.</strong> Apply the promotion and gift credit, charge the balance, and place the order.</td>
<td align="right">6</td>
<td align="right">22.5</td>
<td align="right">41.4 s</td>
<td align="right">166k</td>
</tr>
<tr>
<td align="right">3</td>
<td><strong>Order modification.</strong> Change the jacket size and address while preserving the belt.</td>
<td align="right">3.5</td>
<td align="right">13</td>
<td align="right">29.8 s</td>
<td align="right">95k</td>
</tr>
<tr>
<td align="right">4</td>
<td><strong>Order cancellation.</strong> Cancel fulfillment, refund the payment, and release stock.</td>
<td align="right">3</td>
<td align="right">13</td>
<td align="right">27.2 s</td>
<td align="right">95k</td>
</tr>
<tr>
<td align="right">5</td>
<td><strong>Delivery exception.</strong> Replace only the missing package using the existing investigation.</td>
<td align="right">3</td>
<td align="right">12</td>
<td align="right">26.0 s</td>
<td align="right">95k</td>
</tr>
<tr>
<td align="right">6</td>
<td><strong>Return and exchange.</strong> Exchange the jacket, refund the shoes ($35 to the original card and $15 to gift credit), and keep the belt.</td>
<td align="right">6.5</td>
<td align="right">24</td>
<td align="right">48.0 s</td>
<td align="right">159k</td>
</tr>
<tr>
<td align="right">7</td>
<td><strong>Warranty service.</strong> Book a covered repair after the return window.</td>
<td align="right">3</td>
<td align="right">13</td>
<td align="right">31.5 s</td>
<td align="right">95k</td>
</tr>
<tr>
<td align="right">8</td>
<td><strong>Loyalty account.</strong> Add missing points, reverse duplicates, and keep pending points.</td>
<td align="right">4.5</td>
<td align="right">14</td>
<td align="right">29.9 s</td>
<td align="right">104k</td>
</tr>
<tr>
<td align="right">9</td>
<td><strong>Subscription.</strong> Skip one delivery, change quantities later, and keep the discount.</td>
<td align="right">3.5</td>
<td align="right">6</td>
<td align="right">27.4 s</td>
<td align="right">83k</td>
</tr>
<tr>
<td align="right">10</td>
<td><strong>Stock replenishment.</strong> Account for demand and inbound stock and order the right pack quantity.</td>
<td align="right">4</td>
<td align="right">11.5</td>
<td align="right">26.9 s</td>
<td align="right">109k</td>
</tr>
<tr>
<td align="right">11</td>
<td><strong>Product recall.</strong> Quarantine the recalled stock and initiate customer follow-up without duplicates.</td>
<td align="right">4.5</td>
<td align="right">14.5</td>
<td align="right">37.6 s</td>
<td align="right">102k</td>
</tr>
<tr>
<td align="right">12</td>
<td><strong>Payout reconciliation.</strong> Explain the difference and record the unresolved adjustment.</td>
<td align="right">4</td>
<td align="right">14</td>
<td align="right">34.0 s</td>
<td align="right">99k</td>
</tr>
</tbody></table>
<h3>How we revised and tested the instructions</h3>
<p>For each task, a local runner sent the request and system instructions to the OpenAI Agents API. The agent used the simulated store’s tools and records in a fresh hosted sandbox. The runner handled scripted approvals and saved the results for grading.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/01f461f0-d61a-4b1f-ad28-3e2350c61710.png" alt="The local experiment runner sends tasks to the hosted Agents API. Each run uses a sandbox with one retail tool service and fresh SQLite records. Promptfoo checks the saved results locally." style="display:block;margin:0 auto" />

<p><em>Figure 3. The hosted setup for the twelve distinct retail workflows.</em></p>
<p>We ran the twelve tasks with the baseline instructions and saved the evaluation results, task times, and model costs. A separate model call reviewed the instructions, the recorded runs, and those measurements, then wrote a complete set of revised instructions. The prompt put accuracy first and asked the model to preserve or improve the correctness of the agent’s actions and reports. It also asked for ways to improve speed and reduce cost while meeting the task and approval requirements.</p>
<p>We gave the revised instructions to the retail agent and reran the same tasks with fresh store records. The tools, business rules, approval requirements, and checks stayed fixed. For Revision 2, the model started from Revision 1’s instructions and reviewed the results available from both earlier versions.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/6ed94803-fb21-4973-a4e5-f79d27861b89.png" alt="Four steps for revising instructions. Run the task, check accuracy and record time and cost, revise the instructions in a separate model call, then retest with fresh store records." style="display:block;margin:0 auto" />

<p><em>Figure 4. We applied the same checks before and after each instruction revision.</em></p>
<p>GPT-6.1 Sol ran the tasks at low reasoning effort. The calls that wrote the revisions used the same model at high reasoning effort. The table compares results across instruction versions.</p>
<table>
<thead>
<tr>
<th>Measure</th>
<th align="right">Baseline</th>
<th align="right">Revision 1</th>
<th align="right">Revision 2</th>
</tr>
</thead>
<tbody><tr>
<td>Tasks passed</td>
<td align="right">23/24</td>
<td align="right">24/24</td>
<td align="right">22/24</td>
</tr>
<tr>
<td>Model steps</td>
<td align="right">95</td>
<td align="right">90</td>
<td align="right">86</td>
</tr>
<tr>
<td>Tool calls</td>
<td align="right">321</td>
<td align="right">298</td>
<td align="right">298</td>
</tr>
<tr>
<td>Local command calls, included above</td>
<td align="right">22</td>
<td align="right">0</td>
<td align="right">0</td>
</tr>
<tr>
<td>Total task time</td>
<td align="right">760.6 s</td>
<td align="right">705.1 s</td>
<td align="right">835.3 s</td>
</tr>
<tr>
<td>Estimated task model cost</td>
<td align="right">$1.142–$1.324</td>
<td align="right">$1.053–$1.233</td>
<td align="right">$1.070–$1.248</td>
</tr>
</tbody></table>
<p>Task time runs from submission to the final answer and includes approvals. It excludes setup, grading, evidence collection, and cleanup. The cost estimates use recorded token usage and exclude hosting and the separate requests that wrote the revisions.</p>
<p><strong>Revision 1 cut local work.</strong> The baseline used local commands to create output folders and save reports. The revision told the agent to return its structured answer directly while still writing any required business records. Local command calls fell from 22 to zero, total task time fell by 7.3%, and the upper estimate of model cost fell by 6.8%. Revision 1 passed every check. The baseline correctly changed a subscription but attached the future quantity update to a single delivery cycle in its structured reply.</p>
<p><strong>Revision 2 reduced progress text but took longer overall.</strong> It asked for shorter updates and more concise reasons for actions requiring approval. Compared with Revision 1, progress text fell 24%, yet the agent generated 3% more output tokens and total task time rose 18.5%. In one case, the agent made the right order changes but assigned the belt’s “unchanged” result to the whole order. It also failed the payout check described in the evaluation section. Revision 2 passed both cases when repeated.</p>
<p>Writing the two instruction revisions for the twelve-task study cost an estimated $1.45 and $2.85 in model usage, before the cost of running and checking the tasks. Whether the loop pays for itself depends on how often its useful changes hold up in practice.</p>
<h3>When the reviser kept the instructions unchanged</h3>
<p>In a follow-up experiment, we ran the same twelve retail tasks with GPT-6.1 Terra, a lower-cost model in the same family as Sol. Both models used <code>low</code> reasoning effort. The baseline passed every check on store records and approvals, but three structured reports failed. Revision 1 clarified how to report a verified price, use the exact record ID, and distinguish a new action from one already on record. When we repeated the cases, all twelve passed.</p>
<p>For Revision 2, the reviser considered shortening the reporting guidance and removing some record checks. The guidance addressed the three errors we had observed. The checks served different purposes. Reading a record before an action helped the agent decide what to do. Reading it afterward helped verify what happened. The revision protocol lets the reviser keep the current instructions when the results don’t support another change.</p>
<blockquote>
<p>The revision protocol lets the reviser keep the current instructions when the results don’t support another change.</p>
</blockquote>
<h2>Twelve variations of one task</h2>
<p>The retail return-and-exchange study kept the workflow fixed and varied the customer request. A request covering several items shows how many different decisions can fit into that one workflow.</p>
<h3>Five items, one customer request</h3>
<p>In one example, a customer wanted to exchange a jacket, get refunds for shoes and earphones, and keep a second jacket. They also asked for a refund on an opened game and agreed to take another copy if the store allowed only an exchange. The agent checked the rules for each item, confirmed stock availability, and calculated the refunds before requesting approval. When the payment record changed, it recalculated the earphones refund before issuing it.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/3214b244-0a58-456e-a233-7540a309b24c.png" alt="One recorded retail request progresses through policy checks, refund calculations, approval, and execution." style="display:block;margin:0 auto" />

<p><em>Figure 5. The agent prepares the plan on the left and carries it out on the right. The run used simulated store records and scripted approval.</em></p>
<h3>Comparing the variations</h3>
<p>The twelve variations tested different parts of the workflow. Some asked the agent to identify the right item or handle an unavailable size. Others asked the agent to apply different return policies, check prior refunds, or wait for an inspection.</p>
<p>Revision 1 and Revision 2 both started from Baseline. For each revision, we drafted instructions from one set of examples and compared the versions on a separate set of cases. All tasks passed, including those used to draft the revisions.</p>
<p>The table shows the Baseline averages for each variation. Tool calls include business tool calls and scripted approval calls.</p>
<table>
<thead>
<tr>
<th>Scenario and expected behavior</th>
<th align="right">Avg. model steps</th>
<th align="right">Avg. tool calls</th>
<th align="right">Avg. task time (s)</th>
<th align="right">Avg. total tokens</th>
</tr>
</thead>
<tbody><tr>
<td><strong>1. Mixed returns and exchanges.</strong> Exchange a jacket, refund shoes, and leave another jacket unchanged.</td>
<td align="right">3</td>
<td align="right">9</td>
<td align="right">33</td>
<td align="right">81k</td>
</tr>
<tr>
<td><strong>2. Ambiguous item.</strong> Clarify which of the two matching jackets on different orders the customer means.</td>
<td align="right">6</td>
<td align="right">13</td>
<td align="right">54</td>
<td align="right">106k</td>
</tr>
<tr>
<td><strong>3. Customer changes size.</strong> Update an exchange from medium to large while preserving the customer’s other instructions.</td>
<td align="right">4</td>
<td align="right">12</td>
<td align="right">41</td>
<td align="right">106k</td>
</tr>
<tr>
<td><strong>4. Unavailable size.</strong> Follow the customer’s fallback: keep the jacket if large is unavailable, without issuing a refund.</td>
<td align="right">4</td>
<td align="right">11</td>
<td align="right">43</td>
<td align="right">103k</td>
</tr>
<tr>
<td><strong>5. Split payment.</strong> Refund the original card and gift credit correctly, without moving gift credit onto the card.</td>
<td align="right">3</td>
<td align="right">9</td>
<td align="right">35</td>
<td align="right">81k</td>
</tr>
<tr>
<td><strong>6. Previous partial refund.</strong> Refund only the remaining pair of shoes, without duplicating an earlier refund.</td>
<td align="right">3</td>
<td align="right">11</td>
<td align="right">36</td>
<td align="right">77k</td>
</tr>
<tr>
<td><strong>7. Different return windows.</strong> Complete eligible clothing requests while denying an electronics return outside its allowed window.</td>
<td align="right">3</td>
<td align="right">12</td>
<td align="right">37</td>
<td align="right">86k</td>
</tr>
<tr>
<td><strong>8. Exchange-only item.</strong> Exchange an opened game for the authorized same-title replacement when a refund is prohibited.</td>
<td align="right">3</td>
<td align="right">18</td>
<td align="right">37</td>
<td align="right">112k</td>
</tr>
<tr>
<td><strong>9. Pending jewelry inspection.</strong> Wait for inspection approval before refunding a ring, then complete the eligible return.</td>
<td align="right">4</td>
<td align="right">19</td>
<td align="right">45</td>
<td align="right">143k</td>
</tr>
<tr>
<td><strong>10. Altered jewelry.</strong> Deny a refund for an altered ring, but complete other eligible requests.</td>
<td align="right">3</td>
<td align="right">12</td>
<td align="right">33</td>
<td align="right">87k</td>
</tr>
<tr>
<td><strong>11. Different policy versions.</strong> Apply the policy governing each item, even when similar products have different return rules.</td>
<td align="right">3</td>
<td align="right">17</td>
<td align="right">35</td>
<td align="right">105k</td>
</tr>
<tr>
<td><strong>12. Combined constraints.</strong> Handle a size change, split payment, an expired return, and pending inspection within one five-item request.</td>
<td align="right">4</td>
<td align="right">18</td>
<td align="right">46</td>
<td align="right">128k</td>
</tr>
</tbody></table>
<p>Both revisions cut business tool calls by about a fifth compared with Baseline. Revision 1 raised median task time from 36.8 to 40.0 seconds and the upper estimate of model cost by 1.3%. Revision 2 lowered the median from 38.6 to 36.7 seconds, while the upper cost estimate rose 0.4%. Revision 2 was faster in half of its paired comparisons.</p>
<p>Promptfoo’s comparison view pairs each Baseline run with the revision tested on the same case. That lets us inspect the replies and checks behind the aggregate results.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/e7cd0070-8ec0-44cb-b0f0-b772276fb6f0.png" alt="Promptfoo compares experiment 11 baseline and revised task outputs side by side across both evaluation rounds." style="display:block;margin:0 auto" />

<p><em>Figure 6. Matched Baseline and revised results from the return-and-exchange study.</em></p>
<p>The four examples below show how Baseline and Revision 2 differed by variation. Times are medians, and business tool calls are totals for each version.</p>
<table>
<thead>
<tr>
<th>Return or exchange variation</th>
<th align="right">Median task time, Baseline → Revision 2</th>
<th align="right">Business tool calls, Baseline → Revision 2</th>
</tr>
</thead>
<tbody><tr>
<td>Multiple orders with a previous partial refund</td>
<td align="right">38.9 → 30.0 s</td>
<td align="right">18 → 16</td>
</tr>
<tr>
<td>Two matching items requiring clarification</td>
<td align="right">54.4 → 47.1 s</td>
<td align="right">22 → 14</td>
</tr>
<tr>
<td>Opened item eligible for exchange only</td>
<td align="right">36.1 → 39.5 s</td>
<td align="right">35 → 30</td>
</tr>
<tr>
<td>Jewelry awaiting inspection</td>
<td align="right">42.7 → 48.7 s</td>
<td align="right">34 → 26</td>
</tr>
</tbody></table>
<p>With Revision 2, both the clarification and inspection cases used eight fewer business tool calls. Median time fell for clarification and rose for inspection. That variation matters when deciding whether to adopt a shared instruction set. The study’s rule required every case to pass. A revision also had to lower estimated model cost or median task time without worsening the other. Neither revision met that rule. Revision 2 still reduced median task time, a benefit a team might value differently.</p>
<h2>Framework for applied AI engineering</h2>
<p>The <a href="https://blog.latentthoughts.com/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments">applied AI engineering experiment framework</a> helped us turn a broad claim about self-improvement into tests with measurable outcomes.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/2bda05df-81c3-4bbb-acea-a4f6a8f4fcf9.png" alt="The engineering experiment framework moves from a hypothesis through testing and interpretation to a decision." style="display:block;margin:0 auto" />

<p><em>Figure 7. The eight-step framework used to structure the investigation.</em></p>
<p>The model proposed changes within the experiment. We chose the questions, set the checks, and decided whether the evidence supported continuing.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1c826f09-2586-48a0-8b4e-ebaec57e3907.png" alt="Experiment results feed into interpretation and a decision to revise the investigation or stop." style="display:block;margin:0 auto" />

<p><em>Figure 8. Findings guide the next experiment or a decision to stop.</em></p>
<h2>A revised thesis: continuous adaptation</h2>
<p>The results support the counter-thesis within this workload: a strong baseline left little measured correctness headroom, while some revisions still improved efficiency. That makes the value of an improvement loop a practical question of its benefits versus the effort it requires. How well revisions transfer to unfamiliar requests, and whether gains accumulate over time, remain open questions.</p>
<p>A revised thesis is that the larger opportunity may be <strong>continuous adaptation as requirements change</strong>. A new return policy, a changed service interface, or different customer behavior could create new work for an improvement loop even when the original agent performed well. The tests here varied customer choices within individual tasks. A next experiment could introduce a lasting change to the operating environment and measure whether autonomous revision restores the required behavior, along with the time and cost of doing so.</p>
<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item><item><title><![CDATA[From Hypothesis to Understanding: A Practical Framework for Applied AI Engineering Experiments]]></title><description><![CDATA[Turn a favorable engineering idea into a clear decision: start, keep going, change direction, or stop.]]></description><link>https://blog.latentthoughts.com/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments</link><guid isPermaLink="true">https://blog.latentthoughts.com/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments</guid><category><![CDATA[codex]]></category><category><![CDATA[openai]]></category><category><![CDATA[codex-skil]]></category><category><![CDATA[experimentation]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Tue, 03 Nov 2026 14:28:18 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/c432f4f5-df65-4970-a2cb-a6e6d024082d.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A working demonstration shows what’s possible; an experiment tests whether it’s worth pursuing. Before investing further, we need to know whether the proposed approach solves a worthwhile problem, improves on a competent baseline, and meets the requirements that matter.</p>
<p>I use the following framework most often when working with AI and agentic systems. It turns those questions into testable hypotheses and controlled applied AI experiments, using the results to decide whether to continue, change direction, or stop.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/2bda05df-81c3-4bbb-acea-a4f6a8f4fcf9.png" alt="Eight steps from hypothesis formulation through experimental design, protocol, validation, execution, analysis, interpretation, and conclusion." style="display:block;margin:0 auto" />

<p><em>Figure 1. Eight steps from hypothesis to conclusion. Findings can lead to another experiment, a revised hypothesis, or a decision to stop.</em></p>
<p>You will often need more than one experiment. Each one helps you decide what to test next: run the experiment, analyze the results, adjust the question or design, and test again. The overall framework keeps you focused on the problem, the budget, and what you can reasonably conclude. Agree on the plan before each comparison, then keep it fixed while collecting results. Change the next experiment when the findings call for it.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1c826f09-2586-48a0-8b4e-ebaec57e3907.png" alt="The experimentation loop: execute, analyze, interpret, and decide whether to revise or conclude." style="display:block;margin:0 auto" />

<p><em>Figure 2. The experimentation loop: execute the experiment, analyze the results, interpret the findings, and determine the next step. Findings can lead to a revised hypothesis, design, or protocol for another experiment, or to a conclusion that ends the investigation.</em></p>
<h2>Reusable Codex Skill</h2>
<p>Toward the end of the post, we’ll turn the framework into a reusable Codex skill for designing experiments and reviewing results in your own projects.</p>
<p><a class="embed-card" href="https://github.com/garystafford/engineering-experiments-codex-skill">https://github.com/garystafford/engineering-experiments-codex-skill</a></p>

<h2>The project behind the examples</h2>
<p>We will review a series of recent agent self-improvement experiments to show how it works in practice. The agent self-improvement project explored whether a model could use feedback from a worker’s executions to write instructions that improved that worker’s performance. The initial comparisons across five small exploratory experiments followed the same method: collect baseline results, generate one replacement instruction set, and compare that frozen replacement with the original on held-out cases.</p>
<blockquote>
<p>Our agent self-improvement project explored whether a model could use feedback from a worker’s executions to write instructions that improved that worker’s performance.</p>
</blockquote>
<p>We began with a refund workflow and progressed to multi-turn retail customer service involving orders, payments, inventory, and policy constraints. We checked whether the worker completed the required tasks, attempted any prohibited actions, and used less money or time. The baselines met the business requirements in the completed evaluations, leaving no observed accuracy gap for the revisions to close. Some revisions reduced cost. The investigation did not establish sustained, autonomous self-improvement.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/bc84d5e8-a7f7-42b6-b199-5981dfaeeca2.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 3. Experiment 1 architecture: one refund worker uses a local MCP server backed by a fresh SQLite database for each task. The Python harness collects development traces and grades, requests one replacement instruction set, and compares it with the baseline on held-out cases using an independent deterministic grader.</em></p>
<p>Experiment 1, above, tested a focused refund workflow. The results led us to examine cost more directly and, in subsequent experiments, introduce more complex operational requirements. By experiment 4, the worker coordinated orders, payments, inventory, and support across four simulated services. The framework stayed the same; the findings shaped the next question and workload.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/e06733fe-9d39-4a8d-bbe9-26fa7fbd6588.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 4. Experiment 4 architecture: one worker coordinates four simulated services through local MCP connections, each with its own SQLite database. The Python harness handles API tool calls, restarts and replays tests, and independently grades business effects.</em></p>
<p>Later experiments moved the agent and retail environment into OpenAI-hosted infrastructure. Experiment 9 kept one worker coordinating several retail services, while the experiment harness remained local. The architecture changed; the framework still required controlled comparisons and independent result checks.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/27cc9197-6918-4384-ba69-16e40b9f51ac.png" alt="Experiment 9 architecture: a local experiment harness connects to an OpenAI-hosted agent and retail sandbox." style="display:block;margin:0 auto" />

<p><em>Figure 5. Experiment 9 architecture: the Python experiment harness and independent grader run locally. The worker runs through the OpenAI-hosted Agents API, with retail MCP services and SQLite databases in its hosted sandbox.</em></p>
<h2>The eight-step framework</h2>
<p>The experiments above provide the running example. The eight steps below explain how to formulate a hypothesis, design and execute an experiment, interpret the findings, and decide what to investigate next.</p>
<h3>1. Formulate the hypothesis</h3>
<p>State the thesis you want to investigate, then formulate a testable hypothesis: a specific change should produce a predicted outcome under stated conditions. Identify competing explanations and the results that would challenge your prediction.</p>
<p>Our broader thesis concerned agent self-improvement. We tested a narrower hypothesis: a model-generated revision of instructions could improve task outcomes or efficiency relative to the original instructions. An unchanged outcome or higher cost would challenge the practical benefit of that revision.</p>
<p>Also identify the study’s purpose. An exploratory experiment helps you determine whether an idea works well enough to warrant further investigation. A confirmatory study requires a design and sample size suited to testing a more precise claim. Exploration need not become a full trial if its findings do not justify one.</p>
<h3>2. Design the experiment</h3>
<p>Define the requirements before selecting the implementation: what the system must do, what it must never do, when it should ask for help, and the conditions it needs to handle. With those requirements established, define the intervention, comparison condition, controls, architecture, and evaluation environment. Selecting a baseline means choosing the starting version and measuring its performance during execution. Consider alternatives and explain what the selected design leaves untested. For example, a single-agent experiment does not show how a multi-agent design would perform.</p>
<p>In our experiment 5, the requirements included handling changing customer instructions, multiple orders, split payments, and exchanges. We implemented one worker connected to simulated retail services, using Python, MCP, and SQLite. That exercised a customer-service workflow while leaving multi-agent coordination and production integrations untested.</p>
<p>Separate what you intend to change from what must remain fixed. Otherwise, a favorable result may leave you unable to explain which change helped. In our comparisons, we changed the instructions while holding the model, tools, policies, grader, and execution limits fixed.</p>
<p>Compare against a baseline that already works well. Weakening it to create room for improvement can make the experiment less useful for the engineering decision you actually face.</p>
<h3>3. Specify the protocol</h3>
<p>Turn the experimental design into a protocol that someone else could review and implement. The design defines the comparison; the protocol specifies how to run and evaluate it. Include:</p>
<ul>
<li>The question, your prediction, and the decision the experiment should help you make.</li>
<li>The tasks, the baseline, the proposed change, and what stays the same.</li>
<li>How you will measure correctness, safety, cost, and speed, and which results will be useful.</li>
<li>Which cases you will use, how often you will run them, which you will keep out of development, and how you will reset the test environment.</li>
<li>The tools, estimated cost, spending cap, and limits on each run.</li>
<li>How you will check the test setup, what records you will save, and what the experiment can and cannot tell you.</li>
</ul>
<p>Define exactly when each measurement starts and stops. Task execution, environment setup, evaluation, and reporting answer different questions; measure them separately when the distinction affects the decision.</p>
<p>Explain why the run count and thresholds fit the purpose. A target for deciding whether to continue is different from proof that a result is statistically reliable. A smaller gain can still be a gain, even if it falls short of your target.</p>
<p>Put the limits into the implementation so they actually stop execution. Budget for development and evaluation as well as successful executions. Agree on paid work before running it, and specify how execution stops when limits are reached.</p>
<p>Experiment 5 used 12 conversations to inform the revision and held out 12 new scenario versions to compare against the baseline. Each held-out case ran once per version, with the test environment reset each time, alternating which version ran first. A shared budget cap covered development and evaluation, with separate limits for requests, tool calls, and time. This was enough for an initial comparison, but not to establish how reliably the system would perform.</p>
<h3>4. Validate the experimental setup</h3>
<p>Before running a new experiment, validate the tools, graders, test data, and starting conditions. Unit tests can check individual tools and grading rules; integration tests can check the complete workflow. Confirm that the tasks are feasible, that known correct and incorrect outcomes receive the expected grades, and that each run starts with an isolated state.</p>
<p>Also validate the harness as a measurement instrument. Check how it recognizes completion and failure, handles delayed or missing records, and recovers without repeating completed actions. A harness fault is not evidence that the system under test failed.</p>
<p>In our implementation, the grader inspected actual refunds, exchanges, balances, and stock, as well as logs of prohibited actions. A plausible final answer alone could not pass. A faulty grader could make an effective intervention look ineffective, or vice versa.</p>
<h3>5. Execute the experiment</h3>
<p>Follow the agreed protocol. Keep the model configuration, test conditions, and intervention fixed during evaluation. Collect outcomes, traces, timing, usage, and costs. Record amendments rather than rewriting the original protocol afterward.</p>
<p>Our initial improvement comparisons followed this sequence:</p>
<ol>
<li>Collected baseline execution results, traces, grades, and costs.</li>
<li>Gave the model that evidence and the baseline instructions.</li>
<li>Asked it to generate <strong>one complete replacement instruction set</strong>.</li>
<li>Froze that replacement.</li>
<li>Compared baseline and replacement on fresh held-out cases.</li>
</ol>
<p>That sequence is how we tested instruction improvement. Another investigation might compare models, retrieval strategies, or architectures instead.</p>
<h3>6. Analyze the results</h3>
<p>Calculate the predefined metrics and compare them with the prediction, baseline results, and acceptance criteria. Keep the measurements separate from explanations and adoption decisions. Report worker execution costs separately from development and instruction-generation costs. Look at individual cases, averages, and totals. Separate system failures from test-setup faults, and count the cost of unsuccessful work. Judge the result using the measures you chose beforehand.</p>
<p>Use instrumentation to investigate the suspected cause before changing the implementation. Fewer requests, lower cost, and shorter elapsed time are distinct outcomes. Overlapping model and tool activity also means summed step durations need not equal wall-clock time.</p>
<p>In our investigation, a revision made fewer tool calls but increased cost per successful task. That distinction mattered because the objective was to reduce cost. Another revision produced a modest cost saving but didn’t meet our target for continuing. We needed to report both the savings and that it fell short of our target.</p>
<h3>7. Interpret the findings</h3>
<p>Assess whether the results support the hypothesis, challenge it, or leave it unresolved. Consider alternative explanations, measurement validity, uncertainty, and the conditions to which the findings might transfer. Distinguish functional gains from efficiency gains: meeting a new requirement does not necessarily make a workflow faster or cheaper. Interpretation explains what the measurements mean; the next step states the conclusion and the action that follows.</p>
<p>A small test with simulated tasks can help you decide what to try next. It cannot establish production reliability. A change may perform better without the test explaining why, and testing one architecture tells you little about alternatives. In our investigation, the baselines passed the tested business requirements. We therefore observed no baseline accuracy failures for the replacement instructions to fix. That changed what we wanted to investigate next; it did not disprove self-improvement in general.</p>
<p>Experiment 5 used new data variants within known simulated scenario families, with one execution per case and version. That supports an observation about these cases, not a claim of production reliability or repeatable savings across other workloads. Two successful revision steps also do not establish a sustained or autonomous improvement process. If the results challenge your explanation, change it or consider an alternative. Test that new explanation on fresh cases before making a stronger claim.</p>
<h3>8. Conclude and determine next steps</h3>
<p>Answer the original research question. Within the tested scope, does the hypothesis remain supported, need revision, or warrant rejection? An inconclusive finding is also a conclusion. State that outcome before deciding whether to continue, redesign, investigate an alternative, run a larger study, or stop. A small exploratory study rarely justifies declaring a broad thesis proved or disproved.</p>
<p>Our conclusion was narrower than the original thesis: feedback-driven instruction revision sometimes reduced execution cost while preserving the tested business outcomes. We demonstrated no accuracy gain and did not establish sustained, autonomous self-improvement.</p>
<p>The higher cost of the first experiment prompted a more explicit cost objective in the next. Later experiments added more realistic tasks, including customer conversations with several exchanges. When the experiment-5 baseline passed its accuracy evaluation, we agreed to investigate efficiency. The second revision asked whether another improvement step could reduce cost again while retaining correctness. Each result changed what the next experiment needed to tell us.</p>
<p>A result that falls short of your target may still justify another experiment. Explain what you expect to learn next, why it is worth the additional time and cost, and how it follows from the findings.</p>
<h2>Where this framework fits</h2>
<p>The hypothesis–experiment–analysis–revision cycle is established scientific practice. OpenAI’s <a href="https://developers.openai.com/api/docs/guides/evaluation-best-practices">Evaluation best practices</a> apply that discipline to AI systems through explicit objectives, datasets, metrics, and repeated evaluation. Its <a href="https://openai.com/index/trustworthy-third-party-evaluations-foundations/">shared playbook for trustworthy evaluations</a> further emphasizes matching the evaluation setup to the claim and checking whether the results support it.</p>
<p>Complementary perspectives include <a href="https://arxiv.org/abs/2407.01502">AI Agents That Matter</a>, on cost, holdouts, and reproducibility, and Anthropic’s <a href="https://www.anthropic.com/research/demystifying-evals-for-ai-agents">Demystifying evals for AI agents</a>, on trials, graders, and outcomes.</p>
<p>Research automation has precedents too: Sakana AI’s <a href="https://sakana.ai/ai-scientist/">The AI Scientist</a> connects idea generation, experiments, and reporting; Google’s <a href="https://arxiv.org/abs/2502.18864v1">Towards an AI co-scientist</a> develops and refines research hypotheses.</p>
<p>This framework applies that established cycle to engineering decisions. Before your next experiment, write down the decision it should inform. Afterward, keep your original prediction alongside the result, and explain why the next step follows.</p>
<h2>Put the framework to work as a skill</h2>
<p>To make the framework reusable, we packaged it as an <code>engineering-experiments</code> skill for Codex: written instructions supported by an experiment-brief template, a results-summary template, a decision-record template, and a worked example. The skill follows the same eight steps and offers three entry points: designing an experiment, executing an agreed protocol, or reviewing existing results.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/82d81869-b5a6-4bf9-ab6f-69e35cbef907.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 6. The engineering-experiments skill package:</em> <code>SKILL.md</code> <em>defines the workflow,</em> <code>agents/openai.yaml</code> <em>supplies Codex metadata,</em> <code>assets/</code> <em>contains the experiment templates, and</em> <code>references/</code> <em>holds the worked example. Distributing the complete folder preserves the relative links between these files.</em></p>
<p>Once installed, the skill appears as Engineering Experiments in Codex’s Skills view. Its details show the workflow and supporting resources, so you can inspect the instructions before using it in a conversation.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/983f62d7-4792-4977-ac68-0909be379599.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 7. Engineering Experiments installed and enabled in Codex. The detail view displays the skill’s instructions, design, execute, and review entry points, and links to templates and worked examples.</em></p>
<p>In the desktop app, type <code>@</code> and select <strong>Engineering Experiments</strong>. In the Codex CLI or IDE extension, use <code>$engineering-experiments</code> or select it through <code>/skills</code>. The examples below use <code>$</code>, but both forms select the same skill.</p>
<p>Here, in the Codex desktop app, I ask for a protocol, run count, and budget for a caching experiment, explicitly limiting the task to design. Codex returns a proposal whose assumptions and acceptance criteria I can review and refine before authorizing execution.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/8a7c3fc6-95a4-47d0-b4aa-d749391635a8.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 8. Using Codex’s engineering-experiments skill to design a caching experiment. The response proposes a hypothesis, comparison conditions, and evaluation criteria, and states that it has not executed any runs or paid requests.</em></p>
<p>The same skill is available in the Codex CLI. Here, I ask whether adding a review pass can reduce factual errors enough to justify the extra latency and cost. The request remains at the design stage, allowing me to assess the proposed workload and evaluation effort before committing to execution.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/e0a5b4ae-7240-49ff-b061-d58aec7344ba.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 9. Invoking the engineering-experiments skill in the Codex CLI to plan a review-pass experiment. Codex reads the skill and experiment-brief template, then proposes a run count, API budget, and human evaluation effort. It identifies these as proposed design choices and confirms that nothing has been executed.</em></p>
<p>Using the skill, you can enter at the stage you need rather than restart the whole process. For example, a design request could be:</p>
<blockquote>
<p>Use $engineering-experiments to turn this hypothesis into a bounded experiment brief. Identify the baseline, architecture alternatives, measures, run count, budget, and limits on the conclusions. Flag unresolved choices before execution.</p>
</blockquote>
<p>After collecting results, a review request could be:</p>
<blockquote>
<p>Use $engineering-experiments to review these results against the original brief. Separate observations from explanations, identify what remains uncertain, and recommend whether to stop, revise, or run a fresh experiment.</p>
</blockquote>
<p>The skill’s purpose is practical: help you apply the approach to your next experiment and produce a plan or review you can use. Goals, spending authorization, and the claims made from the results still require human judgment.</p>
<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item><item><title><![CDATA[Single-Agent vs. Multi-Agent Reasoning: Accuracy, Latency, and Cost]]></title><description><![CDATA[Explore when to use single-agent or multi-agent AI workflows, and how to balance accuracy, latency, and cost as you add teams, delegation, and synthesis.]]></description><link>https://blog.latentthoughts.com/single-agent-vs-multi-agent-reasoning</link><guid isPermaLink="true">https://blog.latentthoughts.com/single-agent-vs-multi-agent-reasoning</guid><category><![CDATA[#agent]]></category><category><![CDATA[agentic AI]]></category><category><![CDATA[agentic workflow]]></category><category><![CDATA[Agent-Orchestration]]></category><category><![CDATA[openai]]></category><category><![CDATA[open ai]]></category><category><![CDATA[GPT-6.1 Sol]]></category><category><![CDATA[multi-agent]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Mon, 19 Oct 2026 12:38:33 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/efff6327-db7c-41ba-8bf0-65d3be7fc1e9.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>AI agents combine reasoning with tools to inspect evidence, perform calculations, and make decisions. The promise of organizing them into teams is better decision-making. Specialists can examine different constraints, competing teams can develop complete solutions in parallel using different strategies, and a coordinator can select the strongest candidate or synthesize their findings.</p>
<p>Consider a business choosing which initiatives to fund. A financial analysis may favor one portfolio, an engineering review may reveal a dependency, and an operational check may expose overlapping benefits. Each perspective can change the decision. Bringing them together is where collaboration can earn its value, provided the final plan improves enough to justify the work.</p>
<p>For builders, that creates a practical choice. When should one capable agent handle the whole problem? When would competing teams, specialist collaboration, or synthesis produce a better outcome? Answering those questions means evaluating decision quality alongside the time and computational cost required to deliver it.</p>
<p>In this post, I compare a single agent with four team configurations, keeping the model and nine structured decision problems fixed. All five configurations achieved high accuracy. The single-agent workflow had the lowest average completion time and resource use. This experiment did not establish a general or repeatable accuracy advantage from additional agents. For comparable tasks, the practical takeaway is:</p>
<blockquote>
<p>Optimize for better decisions, not more tokens. Start with one capable agent, and add coordination when measured gains justify the extra time and cost.</p>
</blockquote>
<h2>Five ways to organize AI agents</h2>
<p>My recent posts explored <a href="https://blog.latentthoughts.com/choosing-the-right-openai-model-and-reasoning-level">model and reasoning selection</a>, <a href="https://blog.latentthoughts.com/how-to-set-up-evals-for-a-complex-multi-agent-system">evaluating multi-agent workflows</a>, and <a href="https://blog.latentthoughts.com/beyond-a-single-agent-subagent-delegation-with-gpt-6-1-sol">subagent delegation</a>. Here, the model stays fixed while the organization changes.</p>
<p>Every configuration uses <a href="https://developers.openai.com/api/docs/models/gpt-6.1-sol"><strong>GPT-6.1 Sol</strong></a> with <a href="https://developers.openai.com/api/docs/guides/reasoning#reasoning-effort"><strong>medium reasoning effort</strong></a>. A <em>team</em> means the unit responsible for producing one complete candidate answer. It can contain one agent or several agents working together.</p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Organization</th>
<th>What produces the answer</th>
</tr>
</thead>
<tbody><tr>
<td>V0 · Single agent</td>
<td>One agent, one complete problem</td>
<td>A direct answer</td>
</tr>
<tr>
<td>V1 · Five competing agents</td>
<td>Five single-agent teams, each solving the complete problem with a different strategy</td>
<td>Five candidates, then answer processing</td>
</tr>
<tr>
<td>V2 · Five specialist teams</td>
<td>Five competing teams, each with four predefined specialists, integration, critique, and revision</td>
<td>Five candidates, then answer processing</td>
</tr>
<tr>
<td>V3 · One dynamic team</td>
<td>One coordinator with model-chosen subagents</td>
<td>One integrated answer</td>
</tr>
<tr>
<td>V4 · Five dynamic teams</td>
<td>Five coordinators, each leading a competing team with model-chosen subagents and a different strategy</td>
<td>Five candidates, then answer processing</td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/4d10f9d1-54d5-4f5d-8545-30dc55ab91d1.png" alt="Five configurations showing competing teams and collaboration within each team" style="display:block;margin:0 auto" />

<p><em>Figure 1. Team count and collaboration within a team are separate design choices. Each of the five teams in V1, V2, and V4 attempts the whole problem. The dynamic-team sketches are conceptual; the model chooses actual subagent counts and roles.</em></p>
<p>V1, V2, and V4 follow a hackathon concept: five competing teams tackle the same challenge simultaneously, each pursuing a different strategy. The goal is to compare their complete solutions and select the strongest candidate. Each team works in a separate context and does not exchange findings with other teams during candidate generation. Their errors can still be correlated because they share the same model and evidence. Each strategy guides a complete solution:</p>
<table>
<thead>
<tr>
<th>Strategy</th>
<th>Emphasis</th>
</tr>
</thead>
<tbody><tr>
<td>Inventor</td>
<td>Explore useful novel approaches and alternatives before committing.</td>
</tr>
<tr>
<td>Engineer</td>
<td>Prioritize technical rigor, explicit invariants, and verifiable correctness.</td>
</tr>
<tr>
<td>Minimalist</td>
<td>Prefer the simplest complete solution with minimal implementation complexity.</td>
</tr>
<tr>
<td>Challenger</td>
<td>Challenge assumptions, test adversarial cases, and prioritize robustness.</td>
</tr>
<tr>
<td>Product</td>
<td>Prioritize user value, usability, and practical delivery.</td>
</tr>
</tbody></table>
<p>V1 tests competing teams with one agent per team. V2 adds specialist collaboration within each competing team.</p>
<h3>Predefined specialists</h3>
<p>For each problem, V2 uses four predefined specialist roles. All five competing teams use the same roles for that problem, each guided by a different strategy.</p>
<table>
<thead>
<tr>
<th>Problem</th>
<th>Four predefined specialist roles</th>
</tr>
</thead>
<tbody><tr>
<td>Warehouse queue</td>
<td>Dispatch eligibility; temporal propagation; service audit; evidence review</td>
</tr>
<tr>
<td>Sourcing network</td>
<td>Procurement; inventory operations; commercial finance; evidence review</td>
</tr>
<tr>
<td>Liquidity facilities</td>
<td>Cash reconciliation; treasury operations; financing economics; evidence review</td>
</tr>
<tr>
<td>Revenue cohort bridge</td>
<td>Cohort classification; currency conventions; retention reconciliation; evidence review</td>
</tr>
<tr>
<td>Transformation roadmap</td>
<td>Dependency engineering; benefit analysis; investment finance; evidence review</td>
</tr>
<tr>
<td>Engagement capacity</td>
<td>Skills and independence; capacity operations; commercial finance; evidence review</td>
</tr>
<tr>
<td>Lease fencing recovery</td>
<td>Clock normalization; storage safety; replay recovery; evidence review</td>
</tr>
<tr>
<td>Recovery chain</td>
<td>Data recovery; security and compatibility; recovery operations; evidence review</td>
</tr>
<tr>
<td>Version contract resolution</td>
<td>Protocol compatibility; authorization contracts; product cost; evidence review</td>
</tr>
</tbody></table>
<p>The runner assigns the specialist roles and runs them concurrently. Their notes then feed the builder, followed by critique and revision.</p>
<pre><code class="language-python">roles = self.protocol["organizations"]["V2"]["parallel_specialists"]
notes = await asyncio.gather(*[
    self.stage(
        f"V2-{name}-{role}",
        philosophy=strategy,
        profile=role,
        note=True,
        additions={"role": role},  # Role instructions omitted here.
    )
    for role in roles
])
</code></pre>
<h3>Dynamically organized teams</h3>
<p>The runner launches five competing coordinators under different strategies:</p>
<pre><code class="language-python">teams = await asyncio.gather(
    *(self.dynamic_team(name, strategy)
      for name, strategy in self.protocol["five_strategies"].items())
)
</code></pre>
<p>Inside each <code>dynamic_team</code>, the coordinator starts with native delegation enabled:</p>
<pre><code class="language-python">result = await self.stage(
    f"V3-{name}",
    philosophy=strategy,
    native=True,
    profile="generalist",
    additions={
        "instruction": self.protocol["prompts"]["v3_initial_request"]
    },
)
</code></pre>
<p>In V3 and V4, each coordinator chooses its team’s specialist roles, assignments, and size for that run. The source’s internal <code>V3</code> identifier corresponds to V4 in this article.</p>
<p>This matters because “multi-agent” can describe very different workflows. Comparing five competing single-agent teams with one coordinated team asks a different question from comparing one agent with twenty predefined specialists.</p>
<h3>API, transport, and tool execution</h3>
<p><strong>All five configurations use the</strong> <a href="https://developers.openai.com/api/docs/guides/migrate-to-responses"><strong>Responses API</strong></a> <strong>over</strong> <a href="https://developers.openai.com/api/docs/guides/websocket-mode"><strong>WebSocket transport</strong></a><strong>.</strong> V3 and V4 additionally enable its <a href="https://developers.openai.com/api/docs/guides/responses-multi-agent">Multi-agent beta capability</a>. OpenAI handles spawning, messaging, and coordination; the root agent integrates the results and returns the final answer. The model chooses subagent assignments and team size.</p>
<p>Earlier experiments used the <a href="https://developers.openai.com/api/docs/guides/agents-api/overview">Agents API</a> with hosted sandboxes. I switched this study to Responses because its native subagents could call the same local Python tools as their coordinators. Those tools read evidence, check constraints, validate backup chains, and simulate schedules. The earlier Agents API implementation didn’t support that same <a href="https://developers.openai.com/api/docs/guides/agents-api/multi-agent">subagent function-tool path</a>. Both APIs support delegation.</p>
<p>I chose WebSockets so that the Python experiment runner could return each function’s result as it became available while other agents continued working. All five configurations used the same transport. Multi-agent provides native subagent creation and also supports HTTP.</p>
<h2>Separate exploration from answer processing</h2>
<p>In these workflows, the five completed candidates from V1, V2, and V4 feed three distinct operations:</p>
<ul>
<li><strong>Selection</strong> identifies one existing candidate to deliver.</li>
<li><strong>Synthesis</strong> constructs an answer using all five candidates.</li>
<li><strong>Refinement</strong> checks and revises only the selected candidate.</li>
</ul>
<p>The selected candidate is preserved unchanged; synthesis and refinement produce separate answers. Refinement does not revise the synthesized answer.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1a6c20c7-754a-4a66-880e-d9680fffc950.png" alt="Candidates from five competing teams feed selection and synthesis, while refinement uses the selected candidate" style="display:block;margin:0 auto" />

<p><em>Figure 2. Selection, synthesis, and refinement reuse the same five candidates, so their results are not independent experiments. Complete-workflow totals include candidate generation and all three processing operations.</em></p>
<h3>How synthesis is defined</h3>
<p>Synthesis receives the complete candidate pool, including available work notes, the selector’s choice, and the rationale. It must resolve conflicting findings and assess whether combining them yields a better, more coherent decision. The prompts below expand the stage instructions to make their responsibilities explicit.</p>
<pre><code class="language-python">self.stage(
    f"{mode}-synthesis",
    additions={
        "role": "synthesizer",
        "candidates": anonymous,
        "selection": choice,
        "instruction": """
Produce one complete decision using the original problem, the five candidate decisions and their available work notes, and the selector's choice and rationale.

Your goal is the strongest supported answer under the original requirements, objective, and tie-breakers. Synthesis is useful only when combining findings
improves the result; retaining an existing candidate is acceptable.

1. Assess the candidates. Identify useful findings, supported calculations, missing requirements, and potential errors. Distinguish claims from demonstrated evidence. Agreement among candidates does not establish correctness. Examine the selector's recommendation against the evidence.

...

Return only the complete decision in the original required JSON schema, including the required evidence references. Do not add commentary, evaluation scores, or fields outside the schema. Work without creating subagents.
""",
    },
)
</code></pre>
<h3>How refinement differs from synthesis</h3>
<p>Selection preserves an existing answer:</p>
<pre><code class="language-python">winner = anonymous[choice["winner_index"]]
outcomes["selected"] = winner["decision"]
</code></pre>
<p>The refinement branch receives only that winner:</p>
<pre><code class="language-python">self.stage(
    f"{mode}-refinement",
    additions={
        "role": "winner-only refiner",
        "candidate": winner,
        "instruction": """
Review and improve the selected candidate using its supplied work notes, the original problem, public evidence, and available tools.

Your goal is to make a more accurate, better-supported decision under the original requirements, objective, and tie-breakers. Preserve sound work. Returning the selected decision unchanged is acceptable when further checking supports it.

1. Audit the proposed decision. Check its interpretation of the problem, evidence references, assumptions, calculations, dependencies, and resource constraints. Look for omitted requirements, unsupported claims, double counting, and violations that appear only when the complete plan is considered.

...

Return only the complete final decision in the original required JSON schema, including the required evidence references. Return the entire decision even if only one part changed. Do not add commentary, evaluation scores, or fields outside the schema. Work without creating subagents.
""",
    },
)
</code></pre>
<p>Refinement checks and potentially revises the selected answer, while synthesis draws from the full pool. The runner executes synthesis and refinement concurrently as separate branches.</p>
<p>There are two quality questions here. Does exploring several candidates improve on the single-agent answer? Does combining their findings improve on the strongest candidate already available?</p>
<p>The second question needs a careful comparison. Beating a poorly selected candidate would show useful recovery but would not establish that synthesis produced an answer better than every member of the pool. I compare synthesis with the strongest candidate, identified by deterministic grading after answer generation, with no grading feedback provided to the agents.</p>
<h2>Verifiable decision problems</h2>
<p>Each of the five configurations tackled <strong>nine fixed decision problems five times</strong>, producing <strong>225 completed workflows</strong>. The tasks require more than producing a plausible explanation. They ask for decisions that satisfy interacting rules, calculations, and evidence requirements.</p>
<table>
<thead>
<tr>
<th>Problem</th>
<th>What must hold together</th>
</tr>
</thead>
<tbody><tr>
<td>Warehouse queue</td>
<td>Event order, eligibility, pauses, and delivery timing</td>
</tr>
<tr>
<td>Sourcing network</td>
<td>Procurement eligibility, delivery terms, inventory, and cost</td>
</tr>
<tr>
<td>Liquidity facilities</td>
<td>Settled cash, financing constraints, and financing cost</td>
</tr>
<tr>
<td>Revenue cohort bridge</td>
<td>Customer membership, acquisitions, currency treatment, and retention</td>
</tr>
<tr>
<td>Transformation roadmap</td>
<td>Dependencies, capacity, overlapping benefits, and discounted value</td>
</tr>
<tr>
<td>Engagement capacity</td>
<td>Qualifications, weekly staffing capacity, and margin</td>
</tr>
<tr>
<td>Lease fencing recovery</td>
<td>Clock offsets, leases, storage fences, and safe recovery</td>
</tr>
<tr>
<td>Recovery chain</td>
<td>Backup ancestry, dependencies, and recovery scheduling</td>
</tr>
<tr>
<td>Version contract resolution</td>
<td>Protocol compatibility, authorization, requirements, and migration cost</td>
</tr>
</tbody></table>
<p>These are constructed scenarios with fictional source documents and known scoring rules. The <a href="https://github.com/garystafford/openai-reasoning-benchmark/tree/ed174232282f2079788a9958935fd2da07c0f827">underlying benchmark cases</a> are pinned to a specific revision.</p>
<p>The tools let agents inspect evidence, calculate quantities, and check proposals. These are <a href="https://developers.openai.com/api/docs/guides/function-calling">developer-defined functions</a>: the model requests a call, and the application validates the arguments and permissions, executes the function, and returns the result. The Responses API handles hosted collaboration actions such as spawning and messaging subagents.</p>
<p>V0 receives the full tool catalog. V2 specialists receive role-appropriate subsets, while the integration and answer-processing stages receive the full catalog. V1, all dynamic coordinators, and subagents also have the full catalog. Every workflow has access to the complete task evidence.</p>
<p>Each attempt starts fresh. The runner requests <a href="https://developers.openai.com/api/docs/guides/structured-outputs">Structured Outputs</a> using a JSON schema that specifies required fields and types. Schema compliance is only the first check; it doesn’t confirm the values or decision are correct. Deterministic graders, implemented as code-based checks, assess the requested decision, authoritative evidence, calculations, feasibility, and optimality where required. An answer can be perfectly formatted and still choose the wrong plan.</p>
<h2>High accuracy across configurations</h2>
<p>The single agent returned <strong>43 correct answers in 45 attempts</strong>; the other configurations returned <strong>45 correct answers in 45 attempts</strong>. The single agent solved every repetition of eight problems. Its two misses were both on transformation planning. All answers from the other four configurations were correct.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/7ecc3ae0-8929-4063-b729-917d2d9db9cb.png" alt="Accuracy by configuration and task, with the two single-agent misses isolated to transformation planning" style="display:block;margin:0 auto" />

<p><em>Figure 3. Each cell represents five completed workflows for a single fixed problem. V0 scores 43/45 overall; V1 through V4 score 45/45. For configurations with competing teams, the original winners and the selected, synthesized, and refined answers are equally correct.</em></p>
<p>The observed gap is <strong>4.4 percentage points</strong> and is concentrated in one task. We recorded nine additional single-agent runs, one per problem, separately. All nine were correct, including the additional transformation-planning attempt.</p>
<p>The observed difference is limited to a single task within a small, deliberately selected set. It does not establish a general or repeatable accuracy advantage or formal equivalence between configurations.</p>
<h3>Feasible isn’t always optimal</h3>
<p>The transformation task asks the agent to select initiatives within an <strong>$80,000 capital budget</strong> and <strong>7 units of engineering capacity</strong>. Some initiatives depend on others; two are incompatible. Benefits overlap within beneficiary groups, so adding every listed benefit would overstate value. The objective is to maximize <a href="https://openstax.org/books/principles-finance/pages/16-2-net-present-value-npv-method">net present value</a>, calculated here as discounted future benefits less the upfront investment, using a 10% discount rate.</p>
<p>Two portfolios illustrate why the evaluation must check optimality:</p>
<table>
<thead>
<tr>
<th>Decision</th>
<th>Initiatives</th>
<th>Capital</th>
<th>Engineering units</th>
<th>Net present value</th>
</tr>
</thead>
<tbody><tr>
<td>Feasible, lower value</td>
<td>A, C, E</td>
<td>$60,000</td>
<td>6</td>
<td>$91,239.67</td>
</tr>
<tr>
<td>Exact optimum</td>
<td>A, C, F, G</td>
<td>$65,000</td>
<td>7</td>
<td>$114,173.55</td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1219706b-01db-423b-acde-7f3dfd2b7777.png" alt="Two feasible initiative portfolios and their net present values" style="display:block;margin:0 auto" />

<p><em>Figure 4. Both portfolios satisfy the resource limits. The $22,933.88 objective gap accounts for the two single-agent failures because the task requires the highest-value feasible portfolio. The difference is in value within the fictional task.</em></p>
<p>V0 chose the lower-value portfolio twice and the optimum three times. The multi-agent answers consistently reached the optimum.</p>
<p>This example gives the aggregate accuracy number a concrete meaning. Checking arithmetic and budget compliance would have accepted the lower-value answer. Requiring the best feasible portfolio exposes the decision error. The objective gap is not realized business value or a deployment return-on-investment estimate.</p>
<h2>Workflow time, tokens, and cost</h2>
<p>Timing tells a clearer story than the small accuracy gap. The single-agent workflow had the lowest mean completion time on every problem. Across the nine tasks, it averaged 7.9 seconds; the other configurations averaged 30.1-63.6 seconds.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/acf4defa-e70b-4e58-8399-4c9582358235.png" alt="Mean complete-workflow time for all five configurations" style="display:block;margin:0 auto" />

<p><em>Figure 5. Means cover 45 completed workflows per configuration, with the nine tasks weighted equally. V1, V2, and V4 include all five candidate teams and selection, synthesis, and refinement. V0 and V3 return one direct answer.</em></p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Mean workflow time</th>
<th>Mean reported tokens</th>
<th>Mean estimated workflow cost</th>
</tr>
</thead>
<tbody><tr>
<td>V0 · Single agent</td>
<td>7.9 s</td>
<td>2,687</td>
<td>$0.0065–$0.0071</td>
</tr>
<tr>
<td>V1 · Five competing agents</td>
<td>30.1 s</td>
<td>24,800</td>
<td>$0.0432–$0.0475</td>
</tr>
<tr>
<td>V2 · Five specialist teams</td>
<td>63.6 s</td>
<td>182,431</td>
<td>$0.5226–$0.5748</td>
</tr>
<tr>
<td>V3 · One dynamic team</td>
<td>34.0 s</td>
<td>14,700</td>
<td>$0.0333–$0.0643</td>
</tr>
<tr>
<td>V4 · Five dynamic teams</td>
<td>50.2 s</td>
<td>86,183</td>
<td>$0.2074–$0.3741</td>
</tr>
</tbody></table>
<p>As described earlier, V1, V2, and V4 each produce selected, synthesized, and refined answers. V1 provides an example of the elapsed time and estimated cost as each result becomes available:</p>
<table>
<thead>
<tr>
<th>V1 result</th>
<th>Mean elapsed time</th>
<th>Estimated cost</th>
</tr>
</thead>
<tbody><tr>
<td>Selected answer available</td>
<td>19.4 s</td>
<td>$0.0276–$0.0304</td>
</tr>
<tr>
<td>Synthesized answer available</td>
<td>26.7 s</td>
<td>$0.0369–$0.0406</td>
</tr>
<tr>
<td>Complete workflow, including refinement</td>
<td>30.1 s</td>
<td>$0.0432–$0.0475</td>
</tr>
</tbody></table>
<p>The five-team workflows perform several kinds of work, so their totals should not be read as the minimum cost of every possible multi-agent design. A deployment that stops after selection would spend less than one that also generates synthesis and refinement.</p>
<p>V3 helps examine that distinction. It uses a single dynamic team and returns a single integrated answer, with no external selection, synthesis, or refinement. Its mean time was still 34.0 seconds, compared with V0’s 7.9 seconds. All 45 V3 coordinators delegated, creating 92 observed subagents.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/48227df1-5999-4080-b530-504dca9de7e6.png" alt="Reported token counts and estimated inference cost ranges for complete workflows" style="display:block;margin:0 auto" />

<p><em>Figure 6. Resources cover complete workflows. We estimated costs by applying published model pricing to recorded token usage. The ranges reflect the pricing assumptions used in the cost estimates. Local tool processing and development labor are excluded.</em></p>
<p>Parallel work was enabled. Each dynamic coordinator’s request allowed up to 10 active subagents within its delegation tree, excluding the coordinator itself, and V4 launched its 5 candidate teams concurrently. The setting limits simultaneous activity rather than requiring a particular team size. These timings therefore do not come from a design that required every agent to work sequentially.</p>
<p><a href="https://developers.openai.com/api/docs/guides/latency-optimization#parallelize">Parallel execution</a> can reduce elapsed time when enough work can proceed independently. Assigning work, exchanging findings, and integrating them also take time. Coordination overhead and limited independent work are plausible explanations here, but the experiment does not isolate their causal contributions. Instructions, work allocation, and execution also differ across separate blocks.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/ef3d7359-513f-4cc9-90df-9f60e7fe555a.png" alt="Per-task workflow times showing the single agent as the fastest configuration on each fixed problem" style="display:block;margin:0 auto" />

<p><em>Figure 7. Each row is one fixed problem, with mean complete-workflow seconds across five runs. The single agent has the lowest mean in every row. These observations do not estimate production tail latency.</em></p>
<h2>Cheap winners, costly workflows</h2>
<p>V1’s winning agent looks attractive in isolation. It used an average of <strong>2,506 reported tokens</strong>, compared with <strong>2,687 for V0</strong>, and had a lower estimated inference cost. Its own solve time averaged 8.6 seconds, versus 7.5 seconds for V0.</p>
<p>But the system did not know which candidate to choose before generating the pool. It paid for five attempts, then paid for processing. The full V1 workflow averaged 24,800 tokens and an estimated $0.0432–$0.0475.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/16d69662-3b0e-44a2-a524-54be74420ebc.png" alt="Winning-team cost compared with the cost of the complete workflow" style="display:block;margin:0 auto" />

<p><em>Figure 8. The winning-team ranges include only the team that created the chosen candidate. Complete-workflow ranges include all candidates and processing stages. V0 and V3 each have one team, so the two measurements have the same inference cost.</em></p>
<p>The winning-team measurement shows how the chosen candidate was produced. The complete-workflow measurement includes the resources spent generating alternatives and processing their answers.</p>
<p>This is a question of <a href="https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/">token economics</a>: connecting token consumption to the value of the result. <em>Valuemaxxing</em> asks whether better outcomes justify the resources needed to produce them; <em>tokenmaxxing</em> treats greater consumption as progress.</p>
<blockquote>
<p>Extra reasoning, teams, and checks can be worthwhile when they improve the delivered result enough to justify their cost.</p>
</blockquote>
<p>The accounting boundary should follow the product. If users receive only a selected answer, measure that path. If the application generates alternatives and runs additional checks, retain those resources during system evaluation. Quoting only the winning agent’s cost understates the five-team design’s cost.</p>
<h2>Synthesis tied the best candidate</h2>
<p>Across V1, V2, and V4, <strong>135 candidate pools</strong> contained <strong>675 candidates</strong>. Of those candidates, <strong>671 were fully correct</strong>. Every pool contained at least one correct answer, and the selection delivered a correct candidate from every pool.</p>
<p>Synthesis tied for the highest task-success score in all 135 pools. It improved the primary score in zero pools and reduced it in zero pools. This is an accuracy ceiling. Once one candidate satisfies all task requirements, synthesis cannot earn a higher binary score. It might improve an explanation, but we did not measure explanation quality here. These tasks don’t show how synthesis behaves when every candidate is imperfect.</p>
<p>V1 also shows why we should evaluate competition between teams and collaboration within a team separately. Its five competing single-agent teams produced 221 correct candidates out of 225; each pool retained a correct answer, and selection found one. That is an observed successful workflow. It does not isolate the contributions of strategy diversity, repeated sampling, or specialist collaboration at equal computational cost.</p>
<h2>What to test next</h2>
<p>For a comparable structured decision task, I would begin with a single capable agent, give it tools to retrieve evidence and evaluate proposals, and establish a <a href="https://developers.openai.com/api/docs/guides/evaluation-best-practices">task-specific evaluation set</a> that catches consequential mistakes.</p>
<p>Then inspect failures and add the mechanism most likely to address them.</p>
<table>
<thead>
<tr>
<th>Need in the application</th>
<th>Configuration or mechanism to evaluate</th>
<th>What to measure</th>
</tr>
</thead>
<tbody><tr>
<td>An agent misses a constraint or miscalculates</td>
<td>Better evidence access, deterministic checks, or a focused verifier</td>
<td>The specific error and its recurrence</td>
</tr>
<tr>
<td>Different strategies produce useful alternative solutions</td>
<td>Competing teams solving the whole problem in parallel, followed by selection</td>
<td>Delivered quality and the cost of the whole pool</td>
</tr>
<tr>
<td>The task contains substantial independent work</td>
<td>A coordinator with bounded delegated assignments</td>
<td>Final quality, elapsed time, and total inference resources</td>
</tr>
<tr>
<td>Different roles need different authority or information</td>
<td>Enforced tool, credential, or environment boundaries</td>
<td>Access violations and successful authorized completion</td>
</tr>
<tr>
<td>Useful findings are spread across imperfect answers</td>
<td>Synthesis over deliberately imperfect candidate pools</td>
<td>Improvement over the strongest original candidate</td>
</tr>
</tbody></table>
<p>These are mechanisms to evaluate in follow-up work; this experiment did not measure their benefits. Different authority or information needs may justify multiple agents, but those boundaries require <a href="https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#1-tool-security-least-privilege">enforced tool permissions and access controls</a>. A prompted role alone does not create them. Test both access controls and successful authorized completion.</p>
<p>To test whether agent organization improves accuracy, match the computation budget and control other differences. When choosing a deployment system, evaluate its delivered quality, latency, and cost against the application’s requirements.</p>
<h2>Scope and limitations</h2>
<p>The experiment covers nine deliberately selected, fixed scenarios, five repetitions per configuration, and one model at one reasoning setting. Repetitions reveal variability on those inputs; they do not create additional problem families. Configurations ran in separate execution blocks and used unequal amounts of computation. Three interrupted attempts required fresh replacements; the completed-workflow comparison does not count those interruptions as successful responses or zero-cost work.</p>
<p>The graders assess structured decisions with checkable constraints, leaving open-ended research and writing outside the scope of these results.</p>
<p>For these tasks, <strong>a single tool-using agent was an efficient starting point, and additional organization increased resource usage.</strong> Define what a correct answer must satisfy, measure the full path that delivers it, and add coordination where the evidence warrants it. Judge the extra token spend by the improvement it delivers.</p>
<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item><item><title><![CDATA[Beyond a Single Agent: Subagent Delegation with GPT-6.1 Sol]]></title><description><![CDATA[Learn how GPT-6.1 Sol delegates to subagents through the Responses API, with 120 trials comparing accuracy, speed, and token cost.]]></description><link>https://blog.latentthoughts.com/beyond-a-single-agent-subagent-delegation-with-gpt-6-1-sol</link><guid isPermaLink="true">https://blog.latentthoughts.com/beyond-a-single-agent-subagent-delegation-with-gpt-6-1-sol</guid><category><![CDATA[ai agents]]></category><category><![CDATA[multi-agent]]></category><category><![CDATA[subagents]]></category><category><![CDATA[openai]]></category><category><![CDATA[responses api]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Thu, 08 Oct 2026 03:03:33 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9d6ebbff-8a83-485f-a806-f59a9f5dcd95.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>OpenAI’s GPT-6.1 Sol can delegate work to subagents using <a href="https://developers.openai.com/api/docs/guides/responses-multi-agent"><strong>Multi-agent</strong></a>, a beta capability available through the <a href="https://developers.openai.com/api/reference/responses/overview">Responses API</a>. A root agent coordinates their contributions and combines the results into a final answer. For complex tasks with independent components, such as comparing alternatives, checking calculations, or analyzing different parts of a system, the model can determine where delegation is useful and how to organize the team. This creates opportunities for parallel execution and additional verification, but also introduces coordination overhead. When do those opportunities translate into a better outcome?</p>
<p>Consider reviewing a proposed system change. One subagent could examine correctness, another could check security, and another could look for missing tests, while the root agent reconciles their findings. Giving each a bounded question and separate context can help organize the work and broaden coverage. The same idea applies to comparing vendor proposals or checking separate parts of an operational report. This article follows GPT-6.1 Sol’s subagent capability from delegation and coordination to the final answer. The demo shows how to enable and support that capability, then compares the model working alone with the same model using subagents to see where delegation helps and where its overhead outweighs the gains.</p>
<p>Across twelve constructed tasks with five paired repetitions each, reports with subagents passed every task check in <strong>60 of 60 attempts</strong>, compared with <strong>50 of 60</strong> for a single agent. Mean completion time fell by <strong>6.3%</strong>, while estimated token cost rose by <strong>71.5%</strong>. The accuracy gains were concentrated in three tasks; the study does not establish the same trade-off for production workloads.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9f998a9e-afa7-472e-a424-604fad1c2588.gif" alt="Animated GPT-6.1 Sol subagent delegation: root agent assigns independent work, receives results and combines the final answer" style="display:block;margin:0 auto" />

<p><em>Conceptual example of GPT-6.1 Sol using subagents: the Root Agent delegates independent reviews, receives their findings, and combines them into one final answer. Delegation is optional; the model chooses the assignments and team size.</em></p>
<h2>What GPT-6.1 Sol’s subagents add</h2>
<p>In this demo, GPT-6.1 Sol powers the root agent and every subagent. The <strong>root agent</strong> receives the task and remains responsible for the final answer. A <strong>subagent</strong> receives a delegated assignment and returns its contribution to its parent. Subagents have separate contexts and share the request’s model and available tools, allowing them to focus on bounded questions while working concurrently. If a subagent delegates again, the resulting hierarchy can extend beyond one level. A deeper tree is an organizational choice; its value depends on the work it helps complete.</p>
<p>GPT-6.1 Sol chooses when to spawn subagents, what assignments to give them, when to send follow-up information, and how to combine their results. The Responses API provides hosted collaboration actions that execute those choices. Your application supplies the task, developer tools, and constraints, then checks the final answer. The API handles spawning, messaging, and waiting, while the application executes its developer functions. The root still has to resolve conflicting findings, verify returned work, and synthesize a coherent result, so delegation involves both doing the work and distributing it.</p>
<p>At the request level, <code>multi_agent</code> controls whether the model can delegate. These Python dictionaries show the configuration fragment for each mode, with the single-agent baseline first:</p>
<pre><code class="language-python"># Configuration fragments placed at the top level of a
# Responses request.
single_agent = {"multi_agent": {"enabled": False}}

subagents_enabled = {
    "multi_agent": {
        "enabled": True,
        "max_concurrent_subagents": 3,
    }
}
</code></pre>
<p><code>enabled: True</code> makes subagents available. <code>max_concurrent_subagents</code> limits how many subagents can be active across the whole tree at once, excluding the root. It does not require three subagents or set a delegation depth. Instructions can tell the model when delegation is useful while leaving assignments and team size to its judgment; the complete request and the demo’s exact instructions appear in the implementation section. A small task or a tightly coupled sequence may be easier to complete with a single agent, whereas independent analysis and verification offer more room for useful delegation.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/52cfd0cd-988c-46b2-a543-75ac657899ec.png" alt="Conceptual subagent hierarchy" style="display:block;margin:0 auto" />

<p><em>Figure 1. A possible GPT-6.1 Sol subagent hierarchy. Every blue box is an Agent: the Root Agent owns the answer, and Subagents receive delegated work. Gray arrows show delegation; the dashed path is optional. This is a conceptual example, not an observed team.</em></p>
<p>This distinction matters when judging value. Parallel specialists might reduce elapsed time even when they consume more tokens. A verifier might catch a mistake while adding time and cost. Separate contexts may help a model keep unrelated calculations organized, but the parent still has to reconcile their outputs and honor shared constraints. The potential benefits therefore need to be measured at the final task level. A recorded execution graph establishes that delegation occurred; the final report and its scores establish the outcome.</p>
<h2>Supporting GPT-6.1 Sol’s delegation</h2>
<p>The application supplies the evidence tool and returns its results; the API handles hosted collaboration. Both configurations use <a href="https://developers.openai.com/api/docs/guides/websocket-mode">WebSocket</a>, but their tool-result handling differs: the single-agent configuration continues after a function-call response, whereas the subagent configuration receives results during active execution.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/ac527986-a621-4797-83f1-9cc1f0d32319.png" alt="Matched WebSocket execution arms" style="display:block;margin:0 auto" />

<p><em>Figure 2. GPT-6.1 Sol working alone and with subagents available. Blue boxes identify Agents; gray boxes identify the evidence tool and the final report; and gray arrows indicate control or returned work. Multi-agent mode allows optional Subagents and immediate injection of function results; the single-agent arm uses ordinary continuations on the same socket.</em></p>
<p>The collector keeps subagent contributions separate and retains the root agent’s final report for scoring. The comparison evaluates each complete configuration, including its instructions and execution protocol. The implementation details appear after the results.</p>
<h2>Evaluating GPT-6.1 Sol with and without subagents</h2>
<p>The demo uses 12 tasks across operations, procurement, and planning, with 5 paired repetitions per task, for a total of 120 attempts. Each pair compares GPT-6.1 Sol working alone with GPT-6.1 Sol using subagents, through the Responses API over WebSocket. The model chooses whether to delegate, how many subagents to create, and how deeply to delegate within the resource ceilings. No assignment split, minimum subagent count, or required depth is prescribed.</p>
<p>Both single-agent and subagent-enabled runs use <code>gpt-6.1-sol</code> with medium reasoning, identical source documents, the same read-only <code>read_evidence</code> function and the same root report schema. Sources are supplied through an index of immutable document IDs. The function checks the source hashes and returns only indexed documents; it cannot read private answer keys or arbitrary files. Both arms must return the requested facts, supporting document IDs, findings, final decision, and limitations. JSON validation runs locally rather than relying on an API-enforced root JSON schema.</p>
<p>The arm with subagents enabled enables <code>multi_agent</code> with a concurrency limit of three and adds the discretionary availability paragraph. The single-agent arm disables Multi-agent mode and retains the common task instructions. Both have a ceiling of three levels below the root and a total of 12 agents, including the root. These are maxima, not target sizes: concurrency is an API setting, while depth and total count are instruction ceilings backed by a client stop after an observed violation. They are not hard limits on API creation. The experiment measures this practical configuration as a whole, including its availability instruction and execution protocol.</p>
<p>The documentation examples include delegation instructions, so the demo makes availability explicit while leaving the choice discretionary: “Use native subagents where useful for independent work or verification.” A run that chooses zero subagents remains a valid observation, and neither agent count nor depth earns scoring credit. The comparison measures final outcomes under these two configurations.</p>
<h3>Twelve demo tasks with different reasons to delegate</h3>
<p>The twelve demo tasks cover four workload profiles across three domains. Compact cases create a smaller task in which delegation overhead could dominate. Independent cases offer separable branch, vendor, or hub calculations before a final combined decision. Larger cases increase entity count, source volume, or the size of the optimization space. Uncertain cases require the model to handle missing commitments, incomplete records, or an infeasible combined plan without manufacturing certainty. This study includes all twelve cases, and every case appears in the reported results regardless of which arm performs better.</p>
<p><strong>Operations</strong> cases reconcile orders, fulfillment, returns, credits, and posted invoices against an executed settlement policy. Correct work requires respecting the cutoff, deduplicating ledger records by invoice ID, distinguishing valid split invoices from duplicate billing, and converting local values according to the specified rounding rules. The multi-branch cases provide independent calculations to combine, while the uncertain case contains an incomplete settlement feed. There, some totals must remain <code>null</code>; a partial export is not proof that unobserved billing did not occur. The final decision is whether to hold billing under the policy.</p>
<p><strong>Procurement</strong> cases compare signed vendor commitments and total cost over the stated contract horizon. Discounts apply sequentially, with rounding at each required stage; mandatory integration fees count, and recurring seat, support, and usage charges must be included. Eligibility depends on capacity, uptime, delivery, data location, and contractual recovery commitments. Executed amendments effective by the cutoff can supersede earlier terms; drafts and future changes cannot. The uncertain case has no eligible vendor, so selecting the cheapest option anyway is incorrect, even if its cost is calculated correctly.</p>
<p><strong>Planning</strong> cases combine fork-and-join task dependencies, release dates, access calendars, and a single shared inspector. The objective is to minimize makespan, then the sum of hub opening times, and finally the stated tie-breaking order. Individual hub critical paths can be analyzed separately, but their inspection assignments must be reconciled across the whole plan. The larger case has 1,404 feasible assignments under the reference enumerator. The uncertain case has no feasible complete inspector schedule, so a correct infeasibility decision is more valuable than a plausible-looking date table.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/da7f54d3-71e7-4183-be1a-1e301da6582a.png" alt="Twelve-case design" style="display:block;margin:0 auto" />

<p><em>Figure 3. Twelve demo tasks compare GPT-6.1 Sol with and without subagents across three domains and four profiles. Compact cases intentionally contain no findings; the uncertain cases test whether the model preserves missing information or infeasibility.</em></p>
<p>The demo uses constructed scenarios with known answers to compare accuracy, completion time, and cost. Their deterministic business rules make the comparisons repeatable; they are not records from production organizations or open-ended research tasks. The evidence function reads local immutable records and adds no artificial network delay. The available tool reads evidence; it does not execute code, calculate totals, or solve schedules. Production systems can move exact arithmetic and constraint solving into deterministic tools, thereby creating a different baseline. Consequently, the study tests reasoning, delegation, verification, and synthesis over these source packs; it does not establish the benefit of parallelizing slow external APIs or real-world searches. Keeping that scope clear is essential when interpreting a latency improvement or an accuracy ceiling.</p>
<h2>How we score GPT-6.1 Sol’s final answers</h2>
<p>Before paid calls, the study froze 207 file hashes covering source packs, private answer keys, scoring, runtime, prompts, and the five-repeat schedule. References were calculated directly from the sources and then verified using separately implemented integer arithmetic for accounting and pricing. Planning used a second critical-path and assignment solver to cross-check the optimum and infeasible outcomes. Offline tests also checked perfect reports, deliberately incorrect or unsupported answers, duplicate outputs, hidden-label access rejection, and the collector’s root/subagent and feedback handling. The keys are author-built and algorithmically checked; no independent human label review or paid semantic judge was used.</p>
<p>Quality has several distinct measures. Exact factual accuracy checks requested values, types, and required unknowns. Finding precision and recall distinguish expected issues from invented ones and missed issues; a clean case can earn full credit by reporting no findings. Source checks require the documents needed for each claim, a non-empty explanation, and any prerequisite facts used in the derivation to be exactly correct and source-supported within the same report. The final critical decision is scored separately.</p>
<p>A task passes only when every fact and finding is correct, all source and structural checks pass, the case and decision are correct, and there are no invalid IDs, duplicates, or extra facts.</p>
<p>These source checks assess specified dependencies, not semantic entailment of every sentence or prose quality. The citation-dependency calibration was performed after the runs, using the retained reports and source documents to assess claim-specific source support. It is a post-hoc assessment layer, separate from the pre-run freeze, and does not provide independent validation of new reports.</p>
<h3>Reproducible output checks</h3>
<p>The scoring code is separate from the tool-visible source room. Exact matching distinguishes <code>True</code> from <code>1</code>, permits numeric integer/float matches by value, and preserves JSON <code>null</code> for required unknowns. Finding keys are compared as sets, so precision penalizes invented issues and recall penalizes omitted ones. The source-dependency checker builds on those factual and structural checks; it permits a derived fact to reuse a correct, cited prerequisite within the same report, but does not pool unrelated citations. This is the offline entry point from the distributed code:</p>
<pre><code class="language-python">from citation_calibration import calibrate

# record is one retained run, including its final report.
# This function reads local sources and makes no API calls.
review = calibrate(record)
scores = review["calibrated"]
task_passed = review["calibrated_strict_success"]
</code></pre>
<p>Both arms can see the requested key names and finding formats; expected values and expected issue sets are private. A model can therefore understand the output contract without seeing the answers. The offline validation tests source isolation and both correct and incorrect reports, and the final audit verifies that every recorded initial request still matches the pre-run freeze. This creates an inspectable comparison rather than relying on the model to grade its own answer.</p>
<p>Each demo task receives five matched pairs. The sixty-pair schedule is deterministically shuffled, and within-pair order alternates between subagents-first and single-first. At most two pairs run at once, with sequential arms inside each pair and one fresh WebSocket per assessment. No automatic retries, reconnects, or replacement trials. Failures stay in the all-attempt denominators; completed-report scores and time comparisons show their narrower denominators. Five repetitions estimate variability within a single case rather than treating them as five independent workloads, so overall summaries weight the twelve cases equally.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/968ccaac-24c9-477d-92d8-e656deb2031f.png" alt="Evaluation workflow" style="display:block;margin:0 auto" />

<p><em>Figure 4. Evaluate GPT-6.1 Sol’s delegation choices on the same tasks. Freeze sources, answer keys, and prompts, record the complete execution, then compare final answers, completion, time, and estimated cost. Count and depth describe the chosen work graph rather than contributing to task success.</em></p>
<p>The resource bounds are 600 seconds, twelve responses, and a returned-token observation stop of $0.90 per assessment, with a $25 suite admission allowance and conservative reservations for pending or unknown usage. These are client observation and admission limits, not provider invoice caps. The exploratory benefit screen uses the reported scores: all five pairs must complete with actual subagent work in every run with subagents enabled; mean factual, finding, and source scores and decision and task-pass counts with subagents must be no worse; and the median paired subagents/single time or estimated cost ratio must be at most 0.85. The 15% cutoff is an illustrative screening threshold, not a statistical significance test or a universal definition of value.</p>
<h2>How GPT-6.1 Sol used subagents</h2>
<p>GPT-6.1 Sol produced actual subagent work in <strong>60/60 attempts with subagents enabled</strong>, with <strong>115 returned subagents</strong> across the suite. Each enabled run created 1 to 3 subagents, and the deepest observed level was 1 level below the root. These values came from hosted spawn results and attributed generated work, not from the prompt’s ceilings. The single-agent arm created 0 API subagents. The observed teams used direct subagents of the root. Deeper recursion was available within the stated ceiling, but this suite did not exercise or measure an advantage from a nested delegation chain.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/f6ec484a-7f46-40bd-939d-b4db4c7ae6bc.png" alt="Observed Subagents and depth" style="display:block;margin:0 auto" />

<p><em>Figure 5. The subagent counts and depths GPT-6.1 Sol chose in all sixty attempts with subagents enabled. Filled marks require hosted spawn evidence and subagent-attributed work. Each mark is one repetition; small vertical offsets keep repeated values visible.</em></p>
<p>The enabled flag is only configuration; activation is an execution observation. Hosted <code>spawn_agent</code> results supply returned subagent names, and subagent-generated function calls, assistant output, or collaboration actions demonstrate actual work. The final audit excludes bootstrap user messages from that generation proof. This distinction prevents a planned name or a delivered subagent prompt from being counted as a completed contribution. The retained event stream makes the check possible without executing API collaboration actions in the application.</p>
<p>The recorded operations-independent, pair-one run with subagents enabled illustrates the model’s choices. Subagents named Bayside and Canyon read their respective branch records. The root read Delta’s records and later created a subagent named <code>verify_delta</code>. The root also reread branch records before producing the report. The trace records specialization, verification, and repeated reads of the source. The study does not isolate their individual contributions to time or cost. The example illustrates control flow and is not used as an extra trial or as representative performance evidence for all twelve cases.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9eece691-3eeb-4ac4-bde4-b18e3d250c23.png" alt="Recorded subagent execution" style="display:block;margin:0 auto" />

<p><em>Figure 6. A compact rendering of GPT-6.1 Sol coordinating subagents in one recorded execution from this suite. It illustrates the returned work graph and events; it is not an additional trial or a result selected for superior performance.</em></p>
<h2>Did subagents improve the final answers?</h2>
<p>The two arms delivered <strong>60/60 single-agent reports and 60/60 reports with subagents</strong>. Among completed reports, equally weighted case means were <strong>98.40% single versus 100.00% with subagents for exact facts</strong>, and <strong>100.00% versus 100.00% for finding F1</strong>. Critical final decisions were correct in 60/60 single and 60/60 attempts with subagents enabled. Those separate measures prevent a correct list of numbers from standing in for every aspect of task completion.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/7cb69493-bd34-42a8-af14-dc7792cb8ce1.png" alt="Final report quality and task passes" style="display:block;margin:0 auto" />

<p><em>Figure 7. Final report quality and task pass rates across the 12 demo tasks. Exact facts and findings use the fixed answer keys; source checks use claim-specific dependencies. Task passes count all five attempts per configuration and case.</em></p>
<p>Task success was <strong>50/60 (83.3%) for a single agent and 60/60 (100.0%) with subagents</strong>. The ten single-agent reports that failed contained factual errors; the reports with subagents contained no factual errors. Both configurations achieved 100% on the fact-source and finding-source dependency checks. No completed report failed solely for an unmet citation dependency.</p>
<p>The ten single-agent failures were concentrated in three tasks: operations / large (five failures), operations / uncertain (four), and procurement / large (one). Both configurations passed all repetitions of the other nine tasks.</p>
<p>In operations / large, repetition four, the single agent returned an aggregate expected settlement of $6,906.68, compared with the reference of $6,890.62. Its report matched 32 of 37 facts; the paired report with subagents matched all 37. Both made the correct billing-hold decision, identified every expected finding, and met the source-dependency checks. The incorrect amounts failed the single-agent report; the paired report with subagents passed. This example illustrates a factual difference in the final answers without establishing which part of delegation caused it.</p>
<table>
<thead>
<tr>
<th>Measure</th>
<th>Single agent</th>
<th>Subagents enabled</th>
</tr>
</thead>
<tbody><tr>
<td>Completed reports</td>
<td>60/60</td>
<td>60/60</td>
</tr>
<tr>
<td>Exact factual accuracy, equal task weighting</td>
<td>98.40%</td>
<td>100.00%</td>
</tr>
<tr>
<td>Finding F1, equal task weighting</td>
<td>100.00%</td>
<td>100.00%</td>
</tr>
<tr>
<td>Correct final decisions</td>
<td>60/60</td>
<td>60/60</td>
</tr>
<tr>
<td>Fact-source dependency checks</td>
<td>100.00%</td>
<td>100.00%</td>
</tr>
<tr>
<td>Finding-source dependency checks</td>
<td>100.00%</td>
<td>100.00%</td>
</tr>
<tr>
<td>Task passes</td>
<td>50/60 (83.3%)</td>
<td>60/60 (100.0%)</td>
</tr>
<tr>
<td>Reports containing factual errors</td>
<td>10/60</td>
<td>0/60</td>
</tr>
<tr>
<td>Terminal execution failures</td>
<td>0/60</td>
<td>0/60</td>
</tr>
</tbody></table>
<p>All 120 attempts completed without a terminal execution failure or replacement. Task success and execution completion measure different things: delivering a valid report does not guarantee that its values are correct. The observed completion rate describes these trials and does not establish production reliability or a rare-failure rate.</p>
<h2>Were subagents faster or cheaper?</h2>
<p>The equal-case mean time to the retained valid root report was <strong>85.0 seconds with a single agent and 79.6 seconds with subagents</strong>, a 6.3% reduction. This is the end-to-end assessment latency, including model coordination and source feedback, rather than the sum of subagent compute times. It is conditional on completed reports. Case means and successful-both paired ratios reveal whether that overall result is broad or concentrated in a particular profile, while the five individual ratios retain the observed variability.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/bd58aedf-5907-4810-bf74-e9d91ef223e1.png" alt="Report time and paired ratios" style="display:block;margin:0 auto" />

<p><em>Figure 8. Case mean report times and individual paired subagents/single ratios. Diamonds show medians; a ratio below one favors execution with subagents. Only pairs with two completed reports contribute to the ratios.</em></p>
<p>The mean estimated token cost, with each case weighted equally, was approximately <strong>USD 0.065 per single-agent assessment and USD 0.112 per assessment with subagents enabled</strong>, a 71.5% increase. These estimates are based on returned token usage and have not been reconciled against invoices for the full agent tree. Figure 9 shows the same costs as USD 65.41 and USD 112.17 per 1,000 assessments. The study’s returned-token subtotal was USD 10.6548 across 120 attempts, with complete usage returned for every attempt. An assessment can contain multiple responses and developer function calls.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9f589ae8-5b51-4233-a153-e56d42338890.png" alt="Estimated token cost" style="display:block;margin:0 auto" />

<p><em>Figure 9. Estimated USD per 1,000 known-cost assessments and paired cost ratios. Costs are shown per 1,000 assessments, calculated from the recorded attempts; a single assessment may include multiple responses and tool calls.</em></p>
<p>Cost is based on each unique returned terminal response, priced once at the frozen Standard rates, which still match the model documentation: $2 per million ordinary input tokens, $0.10 per million cached input tokens, $2.50 per million cache writes, and $10 per million output tokens. All 183 unique terminal responses reported <code>service_tier: "default"</code>; the largest returned input count was 74,343 tokens, below the 272,000-token threshold for higher long-context rates. These estimates use Standard pricing without a regional-processing premium. Reasoning tokens are included in output, and subagent-output token estimates are not added again. Missing usage remains unknown. <a href="https://developers.openai.com/api/docs/models/gpt-6.1-sol">GPT-6.1 Sol model and pricing</a></p>
<p>Both arms requested explicit prompt-caching mode without application-supplied breakpoints. The current caching guide says that explicit mode without breakpoints does not use prompt caching or create cache writes. Nevertheless, the retained responses with subagents reported 400,056 cached input tokens across the study; single-agent responses reported none, and neither arm reported cache writes. The estimate prices these returned categories rather than inferring them from the request setting. The records do not explain the origin of those cached tokens or independently reconcile per-agent billing, so the figures are estimates from returned usage, not verified invoice totals for the full subagent tree. <a href="https://developers.openai.com/api/docs/guides/prompt-caching">Prompt caching guide</a></p>
<h2>The trade-offs across all twelve demo tasks</h2>
<p>Pass counts are shown separately for each configuration. Time and cost ratios are medians over five completed pairs per task; values below 1 favor subagents.</p>
<table>
<thead>
<tr>
<th>Case</th>
<th>Facts: single / subagents (%)</th>
<th>Single passes (of 5)</th>
<th>Subagent passes (of 5)</th>
<th>Time ratio: subagents ÷ single</th>
<th>Cost ratio: subagents ÷ single</th>
</tr>
</thead>
<tbody><tr>
<td>Operations / Compact</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>1.263</td>
<td>1.617</td>
</tr>
<tr>
<td>Operations / Independent</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.941</td>
<td>1.960</td>
</tr>
<tr>
<td>Operations / Large</td>
<td>85.95/100.00</td>
<td>0</td>
<td>5</td>
<td>0.821</td>
<td>1.758</td>
</tr>
<tr>
<td>Operations / Uncertain</td>
<td>95.86/100.00</td>
<td>1</td>
<td>5</td>
<td>1.051</td>
<td>2.043</td>
</tr>
<tr>
<td>Procurement / Compact</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.856</td>
<td>1.413</td>
</tr>
<tr>
<td>Procurement / Independent</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.975</td>
<td>1.755</td>
</tr>
<tr>
<td>Procurement / Large</td>
<td>99.05/100.00</td>
<td>4</td>
<td>5</td>
<td>0.865</td>
<td>1.773</td>
</tr>
<tr>
<td>Procurement / Uncertain</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.919</td>
<td>1.819</td>
</tr>
<tr>
<td>Planning / Compact</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>1.251</td>
<td>1.500</td>
</tr>
<tr>
<td>Planning / Independent</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.866</td>
<td>1.458</td>
</tr>
<tr>
<td>Planning / Large</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>0.854</td>
<td>1.363</td>
</tr>
<tr>
<td>Planning / Uncertain</td>
<td>100.00/100.00</td>
<td>5</td>
<td>5</td>
<td>1.023</td>
<td>1.415</td>
</tr>
</tbody></table>
<p><strong>Operations / large was the only task to meet the study’s exploratory benefit screen.</strong> Mean factual accuracy increased from 85.95% to 100%, and task passes increased from 0/5 to 5/5. The median paired time ratio indicated a 17.9% reduction, while estimated token cost increased by 75.8%. Decisions, findings, and source checks were no worse. Operations / uncertain and procurement / large also showed accuracy gains but did not meet the screen’s additional requirement of at least a 15% reduction in time or cost. This illustrative threshold is not a statistical significance test or a complete definition of value.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/cc0696ae-d603-426d-ba28-f60cd00862b2.png" alt="Time and cost trade-offs" style="display:block;margin:0 auto" />

<p><em>Figure 10. Paired time-and-cost ratios, the quality/completion gate, and the exploratory benefit screen using the reported task scores. Lower time or cost alone does not pass the complete screen.</em></p>
<h2>What this means for GPT-6.1 Sol</h2>
<p>For these twelve demo tasks, GPT-6.1 Sol with subagents achieved a 6.3% lower mean report time at a 71.5% higher estimated token cost. Exact factual means were 98.40% for a single agent and 100.00% with subagents, with task passes of 50/60 and 60/60, respectively. Both configurations completed every attempt, identified all expected findings, made all final decisions correctly, and met the source-dependency checks. The model used subagents in every enabled attempt. All observed subagents were direct delegates of the root, so these results do not test the value of deeper delegation.</p>
<p>In this suite, estimated token cost increased for every task, while factual accuracy improved in three. The independent workload profile did not consistently yield substantial time savings. These results support testing delegation against measured errors and completion time on a specific workload, rather than enabling it solely because the work can be divided.</p>
<p>GPT-6.1 Sol’s subagent capability gives it a way to organize independent work and verification before returning one answer. The model chooses the assignments and coordinates the team through hosted collaboration actions; the application supplies tools, constraints, and final validation. This study measured the resulting reports, completion times, returned-token costs, and execution outcomes. It did not measure development effort, maintainability, or human review time. Workloads with slow external tools or separate parts of a real codebase would extend the comparison beyond these local source packs. They would need their own matched measurements: the execution graph shows how GPT-6.1 Sol organized the work, while the root agent’s final answer shows what that organization produced.</p>
<h2>Implementation details</h2>
<p>The following excerpts show how the demo supports hosted delegation, returns developer-tool results, and retains the root agent’s answer. They complement the configuration overview above.</p>
<h3>The connection and the two request modes</h3>
<p>The endpoint, beta header, and location of <code>multi_agent</code> are part of the API contract, not interchangeable names for any multi-agent implementation. This study uses the Responses API beta on a generic <code>websockets</code> connection. Its configured developer tool is only <code>read_evidence</code>; subagent spawning, messaging, and waiting are hosted actions provided by the API. The following configuration excerpts come from the demo runner. They illustrate parts of the complete application: <code>COMMON</code>, <code>DELEGATION</code>, <code>TOOL</code> and the indexed task input are supplied by the project, and the full collector receives events and returns tool results. The connection uses a <a href="https://websockets.readthedocs.io/en/stable/reference/sync/client.html">generic WebSocket library</a> without an OpenAI or Agents SDK.</p>
<pre><code class="language-python">import json
from dotenv import dotenv_values
from websockets.sync.client import connect

key = dotenv_values(".env")["OPENAI_API_KEY"]
headers = {
    "Authorization": "Bearer " + key,
    "OpenAI-Beta": "responses_multi_agent=v1",
}

with connect(
    "wss://api.openai.com/v1/responses",
    additional_headers=headers,
    open_timeout=30,
    close_timeout=2,
    max_size=None,
) as socket:
    socket.send(json.dumps(
        request_body(enabled, initial_input())
    ))
    # The collector receives events and supplies
    # developer-tool results.
</code></pre>
<p>The request builder switches subagent availability together with its discretionary instruction paragraph. The verbatim <code>DELEGATION</code> string preserves the tested wording, including “native” to describe API-provided orchestration; the feature name is Multi-agent. The branch with subagents enabled sets the API’s concurrency maximum; it does not order the model to use three subagents. Depth and total-agent ceilings belong in the common instructions and client observation checks because they are not equivalent to that concurrency field. Here is the request-building logic, with the source/schema input passed in rather than an application-created subagent session:</p>
<pre><code class="language-python">DELEGATION = (
    "Native subagents are available. Use native subagents "
    "where useful for independent work or verification. Choose "
    "whether to delegate, how many subagents to create, their "
    "assignments, and how deeply to delegate within the "
    "resource ceilings. When delegating, give clear questions, "
    "source IDs and expected facts, findings, citations and "
    "limitations. Pass on resource constraints, wait for "
    "results, verify findings and combine them into the final "
    "answer. No agent count, division of work or delegation "
    "depth is prescribed."
)

def request_body(enabled, history, previous=None):
    body = {
        "type": "response.create",
        "model": "gpt-6.1-sol",
        "reasoning": {"effort": "medium"},
        "instructions": COMMON + (
            "\n\n" + DELEGATION if enabled else ""
        ),
        "input": history,
        "tools": [TOOL],
        "multi_agent": {"enabled": enabled},
        "store": False,
        "text": {"verbosity": "medium"},
        "max_output_tokens": 32768,
        "prompt_cache_options": {"mode": "explicit"},
        "include": ["reasoning.encrypted_content"],
    }
    if enabled:
        body["multi_agent"]["max_concurrent_subagents"] = 3
    if previous:
        body["previous_response_id"] = previous
    return body
</code></pre>
<h3>Give the agents a shared tool contract</h3>
<p>The demo declares one <a href="https://developers.openai.com/api/docs/guides/function-calling">developer function</a>, <code>read_evidence</code>, available to the root agent and its subagents. Its schema accepts only indexed document IDs. The local handler checks the allowlist and source hashes before returning the requested documents, ensuring consistent access for every agent. There is no developer-defined spawning function: Multi-agent supplies the hosted collaboration actions separately. This is the actual tool definition, reformatted for readability:</p>
<pre><code class="language-python">TOOL = {
    "type": "function",
    "name": "read_evidence",
    "description": (
        "Read immutable supplied source documents by ID from "
        "the data room index. All agents may call this tool; "
        "it cannot access grading labels or arbitrary files."
    ),
    "parameters": {
        "type": "object",
        "properties": {
            "document_ids": {
                "type": "array",
                "items": {"type": "string"},
                "minItems": 1,
                "maxItems": 36,
            }
        },
        "required": ["document_ids"],
        "additionalProperties": False,
    },
    "strict": True,
}
</code></pre>
<p>The schema describes the function; the application still executes it and returns its output. The handler rejects unknown IDs and arbitrary paths, and neither the schema nor the returned documents expose the private answer keys. This gives subagents the same source access as the root while keeping evaluation data outside the task context.</p>
<h3>Tool feedback is different in the two arms</h3>
<p>Execution with subagents can have an active root and subagents waiting for developer function results, so the collector returns a result immediately when <code>response.output_item.done</code> identifies a completed function call. Each result includes the matching <code>call_id</code>, and multi-agent injection also includes the current <code>response_id</code>. The single-agent response ordinarily finishes at the function-call boundary; its result must instead become input to the next response on the same socket. The following excerpt preserves that distinction and deduplicates calls by ID:</p>
<pre><code class="language-python">if event["type"] == "response.output_item.done":
    item = event["item"]
    if (
        item.get("type") == "function_call"
        and item["call_id"] not in seen_calls
    ):
        result = {
            "type": "function_call_output",
            "call_id": item["call_id"],
            "output": execute_tool(item),
        }
        seen_calls.add(item["call_id"])
        if enabled:
            pending_injections += 1
            socket.send(json.dumps({
                "type": "response.inject",
                "response_id": active_response_id,
                "input": [result],
            }))
        else:
            next_input.append(result)

# This continuation belongs only to the ordinary single-agent
# path.
if (
    not enabled
    and event["type"] == "response.completed"
    and next_input
):
    socket.send(json.dumps(request_body(
        False, next_input, previous=event["response"]["id"]
    )))
    next_input = []
</code></pre>
<p>The continuation is gated on a completed single-agent function-call response. The full collector also decrements pending injections upon acknowledgment, handles documented late-injection input via continuation, records errors, and checks deadlines. It accepts multi-agent terminal completion only after pending acknowledgments have drained, and it retains a valid root report across later responses rather than replacing it with an empty continuation. Those details matter as much as the feature flag: a transport or collection error can hide a correct answer or leave a subagent waiting even when subagent activation itself worked.</p>
<h3>Retain the root agent’s final answer</h3>
<p>The event stream contains contributions from several agents. A subagent’s completed message is useful working material, but the root agent owns the answer returned to the application. The collector uses attribution, completion status, and message phase to select root text before validating it against the report schema. The companion repository contains the frozen runner’s root-message filter. The single-agent path also accepts an unattributed message because ordinary responses can omit agent attribution.</p>
<p>The corresponding helper is available in the <a href="https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public">companion repository</a>.</p>
<p>The full collector deduplicates output items and retains the latest nonempty root report that passes local validation. A later empty continuation therefore cannot erase an already completed answer. Selecting the right author and retaining valid output are separate responsibilities from enabling delegation or delivering function results.</p>
<h3>Verifying subagent activation</h3>
<p>The audit links hosted spawn results to work attributed to the returned subagent names. Bootstrap user messages do not count as generated work.</p>
<p>The corresponding helper is available in the <a href="https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public">companion repository</a>.</p>
<h3>Calculating estimated token cost</h3>
<p>The per-response pricing calculation makes the units explicit. The assessment estimate is the sum over unique terminal response IDs; the per-1,000 figure is that per-assessment mean multiplied by 1,000. Input categories are disjoint, and unavailable usage is not converted into a zero-cost observation.</p>
<p>The corresponding helper is available in the <a href="https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public">companion repository</a>.</p>
<h2>Inspecting and reproducing the comparison</h2>
<p>The distributed project contains the current WebSocket runtime, twelve demo tasks, reference calculations, scoring, and offline tests. In <code>experiments/responses-websocket-suite-12</code>, <code>protocol.py</code> defines common instructions, the read-only tool, and pricing; <code>ws_runner.py</code> implements the event loop; and <code>report.py</code> defines local output validation. <code>grade.py</code> checks exact values, findings, and report structure, while <code>citation_calibration.py</code> supplies the claim-specific source dependencies used for the reported task scores. <code>test_citation_calibration.py</code> checks accepted dependencies and deliberately broken citation chains. The public repository supports inspecting the implementation, running offline tests, and conducting a new paid comparison. Recreating the published figures requires the retained run artifacts, which are not included in the code-only distribution.</p>
<p>These offline tests require no API key and make no API calls:</p>
<pre><code class="language-sh">python experiments/responses-websocket-suite-12/test_suite.py
python experiments/responses-websocket-suite-12/test_citation_calibration.py
</code></pre>
<p>The following commands regenerate the reported comparison only when the retained study results are available; they cannot recreate it from the public code-only distribution alone:</p>
<pre><code class="language-sh">python experiments/responses-websocket-suite-12/analysis/analyze.py
python experiments/responses-websocket-suite-12/citation_calibration.py
</code></pre>
<p><code>analysis/citation-calibration/comparison.json</code> contains the reported task scores, per-case summaries, and unchanged time and cost measurements. The same directory contains per-report source bundles and prerequisite links, as well as preservation checks for the retained run records and the grading audit. Running <code>study.py --freeze</code> and <code>study.py --run</code> would instead create a new paid study and is separate from these offline checks. Both configurations in the reported study used GPT-6.1 Sol with medium reasoning. Higher reasoning effort was not evaluated here.</p>
<h2>Source code</h2>
<p>The <a href="https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public"><strong>repository</strong></a> contains the WebSocket implementation, twelve demo tasks, output contracts, answer keys, reference calculations, deterministic graders, source-dependency checks, and offline tests, so you can inspect how GPT-6.1 Sol’s subagents are configured and compare their results with a single agent on your own tasks.</p>
<p><a class="embed-card" href="https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public">https://github.com/garystafford/hierarchical-multi-agent-systems-demo-public</a></p>

<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item><item><title><![CDATA[How to Set Up Evals for a Complex Multi-Agent System]]></title><description><![CDATA[Build complex agentic evals with Codex and Promptfoo: create datasets, calibrate graders, uncover application bugs, and compare models.]]></description><link>https://blog.latentthoughts.com/how-to-set-up-evals-for-a-complex-multi-agent-system</link><guid isPermaLink="true">https://blog.latentthoughts.com/how-to-set-up-evals-for-a-complex-multi-agent-system</guid><category><![CDATA[Promptfoo]]></category><category><![CDATA[ai agents]]></category><category><![CDATA[multi-agent systems]]></category><category><![CDATA[Model Evaluation]]></category><category><![CDATA[codex]]></category><category><![CDATA[openai]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Tue, 06 Oct 2026 03:13:45 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9b9ba17a-9ccd-4714-adae-3f4470e74eb5.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Evaluating a multi-agent application starts with a practical question: did the system do the job? Answering it takes more than checking the final response. A coordinator might follow a specialist’s incorrect advice. A specialist might produce a sound recommendation from incomplete evidence. Several individually capable agents might combine into a workflow that repeats work, misses a handoff, or declares success too early.</p>
<p>A useful evaluation makes those failures visible and helps you decide what to change. That might be a model or reasoning setting. It might also be a tool, an instruction, or the application’s definition of completion.</p>
<p><strong>The evaluation unit here is an agentic episode:</strong> a task unfolding through decisions, tool calls, returned evidence, and a final answer or change in system state. One test can exercise several dependent steps. The graders inspect properties of that episode: whether the agent retrieved the right evidence, respected its authority and budget, made supported decisions, and reached the required outcome. A simple assertion can detect a consequential failure within that multi-step behavior.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/d65bf344-b861-4dd5-9617-333195e48428.gif" alt="Animated evaluation workflow with a feedback loop: diagnose application defects, harden the app, add regression tests, and recheck before relying on model choices." style="display:block;margin:0 auto" />

<p><em>Figure 1. Start with realistic examples and review expectations. Pilot and comparison results can reveal application defects: diagnose them, harden the application, and add regression tests before revalidating. Keep model choices provisional until the assembled workflow meets its requirements.</em></p>
<h2>Where Codex and Promptfoo fit</h2>
<p>This guide combines a recommended evaluation workflow with observations from the Mars Mission Control experiments. Project examples describe what the harness measured; illustrative examples and recommendations describe how to extend that approach with Codex and Promptfoo. <a href="https://www.promptfoo.dev/docs/intro/"><strong>Promptfoo</strong></a> <strong>is an open-source toolkit for evaluating AI applications and testing them for security weaknesses.</strong> You define test cases, choose the models or application endpoints to exercise, and specify checks called assertions. Promptfoo runs those tests, records outputs and scores, and provides a web viewer for comparing results side by side. Its command-line tool and library let those evaluations become part of a development workflow.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/42e0ccef-5167-4de2-9c32-976feb5bb211.jpg" alt="Promptfoo charts and five model/reasoning columns comparing saved Risk Review results on nine cases." style="display:block;margin:0 auto" />

<p><em>Promptfoo compares configurations on the same cases and lets you inspect the outputs and grades behind each result.</em></p>
<p>OpenAI <a href="https://openai.com/index/openai-to-acquire-promptfoo/">announced an agreement to acquire Promptfoo in March 2026</a>, with plans to continue developing its open-source project. This article focuses on its evaluation capabilities: using a custom adapter to exercise tool-using agents, grade their episodes, and inspect the results.</p>
<p>Codex helps inspect the application, build the recording and replay code, and investigate failures; Promptfoo provides the test runner, assertion framework, and comparison interface. The engineering team still needs to define success, establish evidence of correctness, and decide how the results will support a configuration choice.</p>
<h2>The running example: Mars Mission Control</h2>
<p>The running example is a <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo">five-agent mission simulation</a> built with the <a href="https://developers.openai.com/api/docs/guides/agents-api/overview">OpenAI Agents API</a>. Promptfoo supports its role-level comparisons and a separate view of saved team outcomes. The workflow below applies to other systems that delegate work, retrieve evidence, and coordinate actions.</p>
<p>In this application, the <strong>Mission Director</strong> selects specialists, combines their evidence, and proposes a response for human authorization. <strong>Life Support</strong> assesses cabin conditions and air processing. <strong>Power &amp; Thermal</strong> evaluates electrical reserves and cooling. <strong>Weather &amp; Navigation</strong> assesses hazards, navigation, and crew location. <strong>Risk Review</strong> challenges assumptions, identifies evidence gaps, and flags unsafe trade-offs. These different responsibilities are why each role needs its own cases and grading criteria: a correct resource calculation, a well-supported challenge, and a coordinated plan are different contributions to the same outcome.</p>
<p>The agents work with several kinds of evidence: timestamped telemetry and available recent readings; safety protocols specifying permitted actions, durations, and prerequisites; measured objectives and command execution results; and simulated orbital, maintenance, and crew reports. These reports cross-check the same simulator observations rather than provide independent sensor measurements. Specialists can also use an arithmetic tool to check numerical forecasts, while the Director combines their reports into a proposed plan.</p>
<p>The complexity lies in connecting that evidence under constraints. A proposed crew return may depend on mobility, communications, remaining power, and the hazard deadline. A calculation can be numerically correct but rely on the wrong assumptions or omit a prerequisite. The eval therefore needs to check evidence retrieval, calculations, handoffs, and authority boundaries, in addition to the recommendation. Similar dependencies arise when business agents combine customer records, policy documents, inventory, and tool results to decide what can actually be done.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/2a43c776-059a-4c7e-a4e9-aa9f34ab78a2.jpg" alt="Ares-7 mission dashboard showing an approaching dust storm, the response window, and habitat and crew telemetry." style="display:block;margin:0 auto" />

<p><em>The mission app brings together incident conditions, telemetry, and time pressure for its five agents to assess.</em></p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/ddcff430-4d09-4257-aa5a-29e6fe831674.jpg" alt="Saved mission assessment showing specialist recommendations returning to the Mission Director beside the agent interaction map." style="display:block;margin:0 auto" />

<p><em>In a saved assessment, specialists return evidence-backed recommendations to the Mission Director, who submits a coordinated plan for human review.</em></p>
<h2>1. Map the workflow and define success</h2>
<p>Begin with one representative user task. Follow it from the initial request through the coordinator, specialists, tools, and final outcome. For each agent, record its responsibility, inputs, available tools, expected output, and the agent or component that consumes that output.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/dd7789d6-967b-4820-82a8-91f7ec0357ad.png" alt="Multi-agent workflow with component colors and numbered boundaries for one agent, one handoff, and the entire workflow, including authorization, execution, verification, and failure handling." style="display:block;margin:0 auto" />

<p><em>The system under test. Colors identify component types; numbered boundaries identify evaluation scope. The outer boundary encompasses all components and branches, including authorization, execution, verification, and failure handling. Smaller tests isolate an agent’s episode or a handoff; the coordinator can also be tested as an individual agent. Exercise different paths across separate cases, using controlled approve/decline inputs to test authorization behavior.</em></p>
<p>Then write the success criteria at two levels: what each agent must contribute, and what the complete system must accomplish. These can differ substantially. A research agent may need to return supported claims and citations; the system may need to produce a decision that resolves conflicting evidence. A support specialist may identify the right action; the system still needs to execute the authorized action and verify completion.</p>
<p>Specify important constraints alongside the outcome: permissions, deadlines, tool budgets, required evidence, and acceptable behavior when the request cannot be completed. Include at least one plausible response that should fail. This often exposes ambiguity more quickly than another example of a good answer.</p>
<p>Ask Codex to inspect the real entry points and tool implementations before proposing tests. Check the documentation against the execution path so the tests exercise the intended behavior.</p>
<p><strong>The first deliverable is a workflow map and a written success contract.</strong> Have the person responsible for the product’s behavior review that contract before encoding it in a grader.</p>
<p>Treat this initial contract as a reviewed starting point. Runs across models, prompts, and harness variants can expose missing requirements or valid alternatives. Resolve those observations against domain evidence, version the revised contract, and freeze it for the next comparison. Agreement among candidate outputs does not establish correctness. Cases used to refine the contract are converted to development cases.</p>
<h2>2. Evaluate agents, handoffs, and the complete workflow</h2>
<p>Use three complementary test scopes:</p>
<table>
<thead>
<tr>
<th>Scope</th>
<th>What you exercise</th>
<th>What it helps diagnose</th>
</tr>
</thead>
<tbody><tr>
<td>Individual agent</td>
<td>One role, with controlled input and its permitted tools</td>
<td>Reasoning, retrieval, calculations, and role compliance</td>
</tr>
<tr>
<td>Handoff</td>
<td>A producer’s output as the next agent’s input</td>
<td>Missing information, incompatible structure, lost uncertainty, and conflicting advice</td>
</tr>
<tr>
<td>Complete workflow</td>
<td>The actual agents working together from a user request</td>
<td>Routing, coordination, recovery, resource use, and the final outcome</td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1e0d6d74-7c89-436c-9575-8d997c2ac2e3.png" alt="Three evaluation scopes: individual agents, handoffs between agents, and complete workflows." style="display:block;margin:0 auto" />

<p><em>Figure 2. Each scope answers a different question. Component tests help locate defects; workflow tests establish whether the parts work together.</em></p>
<p>An individual-agent test can still contain multiple steps. The agent might retrieve a record, call a calculator, and then respond. Treat that bounded sequence as one <strong>episode</strong>, and retain its tool activity as well as its answer. Our model-selection comparisons used the following scope: each case involved one agent, its tools, and its final response. A separate end-to-end follow-up, described below, exercised the final selected team on new simulated conditions.</p>
<p>For a fair role comparison, give competing configurations the same starting input. A coordinator can receive a frozen set of specialist reports so its ability to use those reports is measured consistently. Include imperfect or contradictory reports when handling them is part of its job. In our replay, consultation tools returned recorded reports; they did not launch fresh specialists or answer new hypothetical questions. That boundary limits what the component test establishes.</p>
<p>Separately, test the selected agents together. Replayed handoffs isolate a component; newly generated handoffs expose interactions. A collection of passing component tests does not establish that the complete workflow succeeds.</p>
<p>For an illustrative handoff test, suppose a specialist reports that a required reading is unavailable and its recommendation is conditional. The coordinator then presents that recommendation as confirmed. Check whether the specialist identifies the missing evidence, whether the condition survives the handoff, and whether the coordinator obtains the evidence or follows the defined fallback. Judge the meaning of the caveat against the contract, rather than requiring identical wording.</p>
<p>In our experiments, the semantic rubric assessed material uncertainty and unsupported claims. We did not implement a standalone assertion proving that every specialist caveat survived a handoff. Fixed specialist reports controlled the Director’s inputs; they did not constitute a separate scored handoff suite.</p>
<p>In Promptfoo, give each role its own review view and retain a consolidated overview for navigation. A reader should be able to find the relevant cases without sorting through every experiment. Keep phase labels when the tools or grading contract changed.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/7289e589-da7d-4c18-991f-ef56369bb4bd.jpg" alt="Promptfoo evaluation library showing five per-agent views and a consolidated overview." style="display:block;margin:0 auto" />

<p><em>Promptfoo in practice. Five role views each contain nine cases; the overview brings their saved results together. This is an index of staged experiments, not a new combined-team test. Aggregate pass rates are hidden here because configurations cover different roles and phases.</em></p>
<h2>3. Record realistic interactions and build reviewed cases</h2>
<p>Instrument the application before collecting a large dataset. Record at the boundary where requests and tool calls occur so you retain what the agent received rather than reconstructing it later.</p>
<p>For each episode, capture the task, role instructions, model and reasoning setting, tool definitions, tool calls and returned evidence, final output, errors, elapsed time, and available usage. Preserve the initial application state needed for replay. Redact sensitive information and keep credentials out of fixtures.</p>
<p>Use real traces to find representative tasks and difficult interactions. If the project is new, begin with explicitly authored scenarios based on requirements, then add real interactions as the application becomes usable. Include ordinary work, boundary conditions, missing information, infeasible requests, and tool failures. Several repetitions of an easy task do not substitute for this variety.</p>
<p>Turn each selected example into a case with three parts:</p>
<ul>
<li><p><strong>Input:</strong> the task, starting context, state, and tools available to the agent.</p>
</li>
<li><p><strong>Expected behavior:</strong> required outcomes, acceptable alternatives, evidence requirements, and material mistakes.</p>
</li>
<li><p><strong>Provenance:</strong> where the case came from and which application, tool, and rubric versions it uses.</p>
</li>
</ul>
<p>For example, an illustrative support-agent case could start with a customer requesting a refund, a verified order record, and an expired eligibility window. The expected behavior is to explain the applicable policy and use the allowed escalation path. Issuing an unauthorized refund is a critical failure. An agent test checks that decision; a handoff test checks that the coordinator preserves the eligibility constraint; a workflow test checks that the promised escalation actually exists in the resulting system state.</p>
<p>A recorded answer can be wrong. A “golden” dataset needs reviewed expectations grounded in business rules, authoritative sources, checked calculations, or expert decisions. A stronger model can draft reference answers and suggest overlooked cases; its confidence does not certify the labels. Our reference and failure reviews were Codex-assisted development reviews, not independent expert adjudication.</p>
<p>For open-ended work, describe acceptable behavior rather than requiring one exact reference response. A valid alternative plan should pass when it satisfies the same requirements.</p>
<p>Keep some representative cases out of development. Once a case guides a prompt, tool, or rubric repair, it belongs to the development set; it no longer provides untouched validation evidence.</p>
<h2>4. Connect the application to Promptfoo</h2>
<p>Build an adapter, called a <a href="https://www.promptfoo.dev/docs/providers/custom-api/">custom provider</a> in Promptfoo, that invokes the application’s agent execution code. Declare any evaluation-specific substitutions, such as fixed specialist reports or controlled tool responses. The adapter connects a test case to an agent episode.</p>
<p>It should load the fixture, create an isolated session or reset the state, apply the requested model and reasoning settings, execute the agent with bounded time and tool use, and save the result along with its trace. Return enough information for assertions to inspect both the answer and the workflow. A timeout or tool failure should still be recorded as an attempt. Specify whether tools use frozen responses, a simulator, or live test services. Isolate writes from production, and reset mutable state between attempts; otherwise one candidate can change the conditions for the next.</p>
<p>For a multi-agent application, organize the harness around the test scopes above. Role tests supply controlled upstream inputs. Handoff tests exercise the producer–consumer boundary. Workflow tests run the actual orchestration and check the resulting state.</p>
<p>Use a dedicated evaluation directory to keep cases, reviewed expectations, the adapter, assertions, configuration, saved outputs, and reports together. Version the test contract with the application. Check that the requested reasoning setting actually reaches the provider; a configuration label alone is insufficient evidence. The Promptfoo configuration ties together the provider configurations, case inputs, and assertions. Pin the model identifier, supported reasoning setting, and relevant runtime versions with the results.</p>
<p>A simple layout makes those responsibilities visible:</p>
<pre><code class="language-text">evals/
  cases/          # Inputs, initial state, reviewed expectations
  providers/      # Adapters that invoke your application
  assertions/     # Deterministic checks and semantic rubrics
  configs/        # Role, handoff, and workflow comparisons
  results/        # Saved outputs, traces, grades, and usage
  decisions/      # Findings, review notes, and model choices
</code></pre>
<p>Start with one case and one configuration. Confirm that state resets between attempts, the intended agent receives the fixture, the trace contains its actual tool results, and an intentionally wrong answer fails the right assertion. Only then expand the case set and configuration matrix.</p>
<p><strong>Save candidate outputs independently of their grades.</strong> Separate generation and grading stages are useful when calibrating a judge or comparing expensive agent episodes. You can then repair a deterministic check offline or recalibrate a judge against the same answers without regenerating every candidate. Preserve the original grades alongside any regrade.</p>
<p>Grading a recorded tool call does not require re-executing it. Deterministic checks can inspect saved answers and tool activity without additional model calls. Semantic grading adds its own cost, while changes to agent or tool behavior require fresh execution. Our <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/saved-output-provider.ts">saved-output provider</a> retrieves the original answer and tool-audit bytes without starting candidate agents.</p>
<p>For comparison views, use readable case descriptions as rows and clear model/reasoning labels as columns. Keep different roles and changed test contracts distinguishable so a chart does not imply an experiment you never ran.</p>
<p>Open a result cell’s details to inspect <strong>Prompt &amp; Output</strong>, then <strong>Evaluation</strong> for the assertion results. This makes the adapter’s output inspectable alongside its grades.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/2921f34d-7d38-4530-b3db-8723e4dbda63.jpg" alt="Promptfoo Prompt and Output panel showing a case identifier and the saved answer with tool activity." style="display:block;margin:0 auto" />

<p><em>Promptfoo in practice. Our adapter receives a case ID and loads the fixture, so the Prompt field displays that ID instead of the full agent instructions. The saved output contains the answer and tool activity. Preserve the resolved inputs in the trace as well; the display label alone cannot reproduce an episode.</em></p>
<h2>5. Use the simplest grader that can enforce the requirement</h2>
<p>Match the grading method to the requirement. Exact answers and structured extraction may need equality or schema checks. Calculations can use executable checks with justified tolerances. Tool-using workflows need checks on actions, permissions, evidence, and limits.</p>
<p>Promptfoo’s <a href="https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/"><code>javascript</code> assertion</a> supports custom validation. Its <a href="https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/"><code>llm-rubric</code> assertion</a> uses a model to assess written criteria. In our role comparisons, we combined custom deterministic checks with a semantic judge and required both to pass. We fixed the judge at <a href="https://developers.openai.com/api/docs/models/gpt-6-sol">GPT-6 Sol</a> with medium reasoning so changes in the candidate configuration did not also change the grader. The <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/grading-policy.ts">versioned semantic policy</a> uses binary pass/fail scoring and distinguishes unresolved cases requiring review from demonstrated semantic failures.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1aaace14-89b6-4802-b25d-1f9508d1c85b.gif" alt="An episode is checked against executable rules and a semantic rubric, with both required for a complete pass." style="display:block;margin:0 auto" />

<p><em>Figure 3. One possible grading contract: both layers must pass. This illustrative episode passes semantic review but fails a required execution rule.</em></p>
<p>A semantic rubric should identify material errors, required coverage, acceptable uncertainty, and valid alternatives. Keep it scoped to the role and actual request. A specialist should not have to reproduce the coordinator’s entire plan.</p>
<p>Keep diagnostic findings distinguishable for evidence retrieval, delegation, handoff integrity, answer quality, and resource use. Separately define which failures block an overall pass. An unauthorized action should cause the episode to fail, regardless of answer quality; a redundant lookup can be evaluated against a declared efficiency requirement. These are contract decisions, not weights to invent after seeing the results.</p>
<p>Our <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/assertions.ts">JavaScript grader</a> returns a single pass/fail result with a reason, alongside a separate semantic judgment. It checks explicit requirements such as schema validity, evidence consultation, lookup limits, and matching plan submission. We did not report separate numeric scores for delegation and evidence gathering.</p>
<p>Decide how to combine the checks before comparing models. In our project, an answer passed semantic review but exceeded a declared tool-call limit. Its complete episode failed because the selection policy required compliance with that limit.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/ddd4c5cc-2f9a-48be-8906-c237859bd7d1.jpg" alt="Actual Promptfoo assertion detail showing a failed workflow check and a passing semantic judgment." style="display:block;margin:0 auto" />

<p><em>Project example. The two graders reveal different properties of the same saved episode. A single overall score would hide the reason for its failure.</em></p>
<h2>6. Calibrate the judge and investigate failures by layer</h2>
<p>Before using scores to select models, test the grader on a small set of reviewed examples: clear passes, clear failures, and borderline cases. Keep the judge configuration fixed, inspect any disagreements, and determine whether the problem lies in the candidate answer, the reference label, or the rubric.</p>
<p>A different judge model does not establish independence or eliminate shared errors. For a stronger validation process, use separately reviewed examples, include plausible but incorrect answers, and inspect false acceptances and false rejections. Randomize presentation order when judging alternatives side by side. These are recommendations; our development calibration did not establish independence from the assistant that helped develop the grader.</p>
<p>The judge also needs an accurate account of the candidate’s evidence. Distinguish facts the agent received, facts it could retrieve, and facts available only to the evaluator. Missing a required retrieval can be an agent failure. Assuming the agent already saw a fact it never received can be a grading failure. Additional evaluator evidence can legitimately verify correctness or outcomes; the rubric must distinguish that use from claims about what the agent knew.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/08e97de5-1b67-4984-ac11-3acb14f5f330.png" alt="Two evidence paths from one source: a filtered tool result reaches the agent, while broader context reaches the judge." style="display:block;margin:0 auto" />

<p><em>Figure 4. A general example of unequal evidence. Check the grounding of a claim against what the agent received, and evaluate required retrieval separately. Broader evidence can still establish factual correctness or the final outcome; it should not be treated as something the agent already knew.</em></p>
<p>We encountered this in our five-agent application: a tool filter omitted relevant readings, while the judge received broader context. Investigating the trace exposed a tool defect and a grading assumption. Changing the candidate’s reasoning setting would leave both defects in place.</p>
<p>Other failures exposed an incomplete definition of success. The application could recognize an intermediate recovery before the required outcome was complete. Similar mistakes are possible when a workflow treats “request submitted” as “task completed.”</p>
<p>For each failure, record the requirement, the supporting evidence, and the layer that needs attention: model, instructions, tool, orchestration, or grader. Codex is useful here because it can follow the trace into the implementation and reproduce many defects offline.</p>
<p>Be especially careful when a regrade improves scores. The same saved outputs went from 43 to 99 complete passes out of 135 after grading corrections. The answers were unchanged. This was evidence of a changed measurement, not improved model capability. Agreement on examples used to develop a judge also does not establish its accuracy on new cases. Our 27-example pilot used Codex-reviewed development labels and one judgment per selected saved output. It did not establish accuracy on independent cases or consistency across repeated judgments. For higher-stakes decisions, have domain reviewers assess disagreements and check the frozen judge on separate cases. The <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/report-judge-calibration.ts">calibration reporting code</a> explicitly records these limitations.</p>
<p>In our regrade, a provider cache reused 27 earlier judgments despite a CLI-level cache setting being in place. Those valid results saved cost, but they were not fresh observations of judge consistency. Record whether each answer and judgment was generated, retrieved locally, or billed with provider-side token caching; these are different forms of reuse.</p>
<h3>When tools or graders change</h3>
<p>Use the following change rules as a recommended workflow:</p>
<table>
<thead>
<tr>
<th>Change</th>
<th>What to do</th>
</tr>
</thead>
<tbody><tr>
<td>Assertion or rubric</td>
<td>Regrade saved episodes that contain the required evidence; preserve original grades and review any disagreements. Collect new episodes if the required evidence is absent.</td>
</tr>
<tr>
<td>Judge model or prompt</td>
<td>Recalibrate using the reviewed examples before comparing scores with the new grader.</td>
</tr>
<tr>
<td>Tool schema, returned evidence, or behavior</td>
<td>Run fresh episodes against the changed application and review affected expectations.</td>
</tr>
<tr>
<td>Success criteria</td>
<td>Create a new contract version and label comparisons accordingly.</td>
</tr>
</tbody></table>
<p>Our <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/regrade-config.ts">regrade configuration</a> binds saved outputs and rubrics to reviewed artifacts using hashes and rejects stale rubric bindings. That controls changes within a comparison. It does not demonstrate ongoing monitoring of judge drift. Even a fixed judge needs renewed validation when the types of answers or evidence it receives change materially.</p>
<h3>What these problems mean for a new project</h3>
<p>The investigation exposed several assumptions to address when building an evaluation from scratch:</p>
<table>
<thead>
<tr>
<th>What we encountered</th>
<th>Real-world consideration</th>
<th>Build this into the first eval</th>
</tr>
</thead>
<tbody><tr>
<td>Recorded responses and stronger-model references still needed review</td>
<td>A plausible answer can violate a refund policy, an engineering constraint, or a business rule</td>
<td>Separate observed answers from reviewed expectations; document valid alternatives</td>
</tr>
<tr>
<td>A tool omitted facts that the judge could see</td>
<td>Retrieval filters, permissions, or stale records can make an apparent reasoning failure an evidence problem</td>
<td>Record the exact tool results received and tell the judge which facts were available</td>
</tr>
<tr>
<td>The simulator could score intermediate recovery as success</td>
<td>Creating a ticket, submitting a job, or proposing an action does not establish completion</td>
<td>Assert the required final state and any observation window, not just the agent’s claim</td>
</tr>
<tr>
<td>Regrading unchanged answers changed the leaderboard</td>
<td>The measurement can improve while the model stays exactly the same</td>
<td>Calibrate against reviewed examples, version the rubric, and retain original grades</td>
</tr>
<tr>
<td>An answer passed semantic review but failed a tool-use limit</td>
<td>Retries and extra lookups can violate latency, cost, or operational constraints</td>
<td>Grade answer quality and workflow compliance separately; justify the limits before selection</td>
</tr>
<tr>
<td>Saved results were initially awkward to compare in the viewer</td>
<td>Valid experiments can still produce a misleading report, especially across different test contracts</td>
<td>Design readable case-by-configuration views early; label phases and preserve errors and untested cells</td>
</tr>
</tbody></table>
<p>Treat a failed test as the start of a diagnosis. Use the trace to locate the defect before changing the model. Keep a findings log with the evidence, affected layer, correction, and whether the case or grading contract changed. This makes the evaluation useful for both improving the application and selecting models.</p>
<h2>7. Harden the application when the eval exposes execution weaknesses</h2>
<p><strong>Building an eval can reveal that the application needs work before model selection is the main decision.</strong> Realistic episodes exercise retrieval, tool validation, handoffs, shared state, deadlines, and recovery. A leaderboard can look encouraging while those parts of the application remain unreliable. Treat the selected models as provisional engineering choices until the assembled system has demonstrated the behavior you require.</p>
<p>Make application hardening a standard part of the feedback loop in the evaluation process. A failed episode should lead to a diagnosis: did the model reason poorly, did a tool return incomplete evidence, did orchestration mishandle a failure, or did the grader measure the wrong thing? <a href="https://www.promptfoo.dev/docs/tracing/">Promptfoo’s tracing support</a> helps inspect the actions behind a result. Budget time for this loop when the application has several agents, external tools, and stateful execution.</p>
<p>For example, our application concealed useful tool-validation details behind a generic error and propagated a specialist timeout without reliably preserving completed reports for review. We improved the feedback and made the failure policy explicit: a requested specialist that fails blocks the proposal, while completed reports remain available for diagnosis. Cancellation must be followed by checks to confirm that the remote work has stopped. Human authorization remains required for every executable proposal. Offline regression tests exercise these behaviors. They do not establish that a live agent will recover correctly or that the remaining evidence is sufficient for every decision.</p>
<p>This work can slow an initial model comparison because some apparent model failures turn out to be application defects. It also produces tests for weaknesses that happy-path demonstrations missed. In a support workflow, a specialist timeout might leave a refund policy unchecked; in an operations workflow, a completed subtask might be lost when another service fails. Define which missing assessments permit a limited proposal and which must block it. A stronger model does not automatically repair either contract.</p>
<p>Keep the loop bounded: reproduce the defect, fix the responsible layer, add an offline regression, and version the changed contract. Reuse saved outputs when investigating grading, but validate execution changes against the changed application before claiming improved reliability. Resolve defects that undermine measurements or required behavior; record lesser issues in the acceptance criteria. This is a stopping rule for a useful engineering phase, not a requirement to eliminate every imperfection before learning from an eval.</p>
<h2>8. Compare configurations under a fixed decision policy</h2>
<p>Define how quality, latency, and cost determine a choice. One product might prefer the cheapest configuration that meets the quality threshold. Another might prioritize complete success, then use latency to break ties. Record critical failures separately so an average cannot conceal them.</p>
<p>Start with a small pilot to verify the harness and observe duration and usage. An episode may contain several model and tool calls, followed by a judge call. Estimate the total workload from those observations rather than multiplying the number of test rows by the price per response. Report missing usage as unknown. Our long-running episodes made elapsed time a poor proxy for token spend. Set concurrency, timeouts, retry limits, and a spending boundary explicitly, then report duration and measured usage separately.</p>
<p>When an evaluation ends a session, stop the work, reconcile usage, save any unresolved accounting evidence, and then delete the session; deleting first can remove the evidence needed to finish accounting.</p>
<p>For future comparisons, report cost per successful workflow alongside cost per attempt. Divide the total execution cost, including failed attempts and retries within a defined retry policy, by the number of workflows that meet the success contract. Report evaluation and judge costs separately. With no successes, the metric is undefined; with incomplete accounting, label any known-cost subtotal as partial. We did not use this metric to select the reported team.</p>
<p>For a system with several agents, compare plausible configurations per role before exploring whole-team combinations. Exhaustively testing every combination quickly becomes expensive. Use the role results to narrow the choices, then validate the proposed team together. Agent behavior can vary even with identical inputs. Add repetitions appropriate to the decision’s risk, report the variation, and keep failed attempts visible.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/51620678-f4aa-4471-bc1c-9390ed9b7c8b.jpg" alt="Actual Promptfoo view comparing three reasoning settings for one agent, with saved results and native charts." style="display:block;margin:0 auto" />

<p><em>Project example. Mission Director passed 7/9 cases with low reasoning and 8/9 with both medium and high reasoning. The charts summarize this role’s saved results, not the reliability of the entire system.</em></p>
<p>Read down a column to find a configuration’s weaknesses; read across a row to compare how configurations handled the same case. Use the failure filters and cell details to explain a difference before interpreting an aggregate score.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/812b0e2b-b030-45c8-96ec-ae5efc62ea83.jpg" alt="Promptfoo comparison of GPT-6 Sol and Astra at medium reasoning on the same nine Power cases." style="display:block;margin:0 auto" />

<p><em>Project example. A focused comparison keeps the case set and the level of reasoning fixed.</em> <a href="https://developers.openai.com/api/docs/models/gpt-6-sol"><em>GPT-6 Sol</em></a> <em>passed 8/9 complete episodes, and Astra passed 9/9. The visible failing cell identifies an evidence-lookup limit, connecting the model choice to a declared workflow requirement. These are saved final grades, not a fresh run.</em></p>
<p>We applied <a href="https://developers.openai.com/api/docs/models/gpt-6-luna">GPT-6 Luna</a> at medium reasoning for Mission Director, Life Support, and Weather &amp; Navigation, and <a href="https://developers.openai.com/api/docs/models/gpt-6-astra">GPT-6 Astra</a> at medium for Power &amp; Thermal and Risk Review. Our policy prioritized complete passes, then median episode time on ties, then lower reasoning effort. Incomplete original candidate usage limited cost conclusions; those results did not establish a cost winner. All five selections met an exploratory gate of at least eight complete passes out of nine in their respective role comparisons.</p>
<p>Nine related development cases per role supported provisional engineering choices, not statistical superiority or a validated, reliable system. The later execution findings require application hardening and separate validation; they do not retroactively change those saved comparison scores. Higher reasoning did not consistently improve results within these limits.</p>
<p>Verify access through the exact endpoint and credentials the harness uses. We initially confused a Codex-session model-access error with an error from the application API; those were separate paths. A tiny authorized smoke test can distinguish access problems from candidate failures before launching a matrix.</p>
<p>Set a stopping rule before the final comparison: resolve known measurement blockers, freeze the contract, run the agreed matrix, and make the decision. New observations belong in a findings log. They should lead to another experiment when they threaten the decision or reveal a meaningful product risk, rather than automatically expanding every run.</p>
<h3>Check the assembled system and the path it takes</h3>
<p><strong>Evaluate the assembled system after selecting its components.</strong> Strong individual-agent scores do not establish that the workflow will succeed. Exercise real handoffs, shared deadlines, and recovery behavior on new cases before drawing conclusions about the team.</p>
<p><strong>Grade outcomes and execution behavior separately.</strong> Check both whether the task was completed and whether the agents followed the required tool-use and authorization rules. A successful outcome can coexist with invalid tool calls or weak recovery; neither an outcome score nor a trace score tells the whole story.</p>
<p>Our mission follow-up illustrated both lessons: a specialist timeout stopped one assessment, while other missions reached their objectives despite rejected calculator calls. We kept those findings separate from the existing model rankings.</p>
<p>The assembled-team follow-up used automated approval of each exact proposal and deterministic checks of mission outcomes. The <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/evals/promptfoo/report-final-team.mjs">outcome report</a> checks mission resolution, objectives, hard safety failures, and exact-proposal authorization, without new semantic judge calls. It exercised the application’s authorization path but did not evaluate human review decisions or independently assess each agent’s rationale. Passing these checks alone does not prove that every unauthorized execution path is impossible.</p>
<h2>9. Give Codex a bounded evaluation task</h2>
<p>Codex can help map agent boundaries, build recording and replay, draft cases, implement assertions, and trace failures back to their source. Its advantage is its ability to work across these connected artifacts. Keep product policy and consequential label disagreements subject to domain review; having the same assistant build and review a grader does not make that review independent.</p>
<p>A useful starting request is:</p>
<blockquote>
<p>Inspect this multi-agent application and map its agents, handoffs, tools, and final outcomes. Propose success criteria and tests at the individual-agent, handoff, and complete-workflow levels. Build a small Promptfoo harness using the application’s execution path, with isolated sessions and saved outputs. Keep observed responses separate from reviewed expectations. Add deterministic checks and a semantic rubric where needed, and document ambiguous labels or application defects. Prepare a pilot plan with estimated cost and duration for approval before paid execution. Preserve outputs so grading can be revised without rerunning candidates.</p>
</blockquote>
<p>Review the workflow map, a few cases, and the grading rules before authorizing a broad run. This gives you concrete decisions to make early, when corrections are inexpensive.</p>
<p>The first milestone is a small evaluation package with explicit inputs, reviewed expectations, inspectable traces, and a documented selection rule. Add reviewed failures as regression cases, and rerun the relevant suites when prompts, tools, models, or policies change. Keep fresh validation cases separate from the examples used to make those repairs.</p>
<h2>Source code</h2>
<p>The <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo"><strong>repository</strong></a> contains the Mars Mission Control application, a custom Promptfoo provider, deterministic checks, a semantic grading policy, calibration and reporting code, and a selected model configuration. The public code-only distribution excludes recorded runs, reviewed datasets, reference answers, calibration labels, and saved results. You can inspect the implementation and adapt it to your own cases, but reproducing the published scores requires the study artifacts that were excluded from the publication. See the <a href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo/blob/a2e9cf61f15bfcbc96f30a434bce18bc9a13c5cd/README.md">README</a> for setup requirements.</p>
<p><a class="embed-card" href="https://github.com/garystafford/openai-agents-api-mission-simulation-demo">https://github.com/garystafford/openai-agents-api-mission-simulation-demo</a></p>

<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item><item><title><![CDATA[Choosing the Right OpenAI Model and Reasoning Level]]></title><description><![CDATA[Benchmarking GPT-6's reasoning across multiple complex tasks to reveal trade-offs among accuracy, latency, and cost.]]></description><link>https://blog.latentthoughts.com/choosing-the-right-openai-model-and-reasoning-level</link><guid isPermaLink="true">https://blog.latentthoughts.com/choosing-the-right-openai-model-and-reasoning-level</guid><category><![CDATA[openai]]></category><category><![CDATA[Model Evaluation]]></category><category><![CDATA[GPT6]]></category><category><![CDATA[GPT-6]]></category><category><![CDATA[GPT-6 Astra]]></category><category><![CDATA[model-reasoning]]></category><dc:creator><![CDATA[Gary Stafford]]></dc:creator><pubDate>Fri, 02 Oct 2026 01:46:46 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/7756c8ca-1355-4659-ae78-072c8451a737.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Your AI application needs answers it can trust, delivered fast enough and at a cost that makes sense. Choosing the right model and reasoning level means defining those requirements and testing which configurations meet them.</p>
<p>In <a href="https://garystafford.medium.com/the-right-model-and-reasoning-level-depend-on-the-workload-482a33d1d644">my earlier post</a>, I compared models across multiple providers. This post compares GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna on 27 constructed reasoning tasks, using three evaluation sweeps run on October 1, 2026. The main comparison uses <code>gpt-6-sol</code>, not the newer <code>gpt-6.1-sol</code>; a separate exploratory example uses GPT-6.1 Sol.</p>
<p>Across these tasks, <strong>Luna/max was the least expensive configuration meeting the default selection gate</strong>, while <strong>Sol/high had the lowest observed median latency among qualifying configurations</strong>. Repeated attempts exposed errors that the first sweep missed, changing which configurations qualified. These are benchmark observations, not production reliability estimates.</p>
<h2>Define what success means</h2>
<p>Evaluate against the requirements of your application:</p>
<table>
<thead>
<tr>
<th>Reason to evaluate</th>
<th>Question to answer</th>
</tr>
</thead>
<tbody><tr>
<td>Accuracy</td>
<td>Does it get your tasks right?</td>
</tr>
<tr>
<td>Latency</td>
<td>Is it fast enough for the application?</td>
</tr>
<tr>
<td>Cost</td>
<td>Is the quality worth the cost per task?</td>
</tr>
<tr>
<td>Consistency</td>
<td>Does it succeed across repeated attempts?</td>
</tr>
<tr>
<td>Workload fit</td>
<td>Which model and reasoning level works best for each use case?</td>
</tr>
</tbody></table>
<p>This post focuses on accuracy, latency, and cost, using repeated attempts and different domains to examine consistency and workload fit.</p>
<p>An interactive application may need a quick answer; a background planning job may take longer to produce a better plan. A small error may be acceptable in an estimate and unacceptable in a release gate.</p>
<p>A key part of defining requirements is <strong>deciding what counts as correct and making sure the grader measures it fairly</strong>.</p>
<p>Does a correct schedule need to be feasible or optimal? What units and rounding rules apply? Does list order matter? Define these requirements before looking at model scores.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/c7b39d21-215c-4b0b-ab17-24ed4e4bbe04.png" alt="Define accuracy, latency, and cost requirements" style="display:block;margin:0 auto" />

<p><em>Figure 1. Keep the requirements short enough to use and precise enough to grade.</em></p>
<p>Specify whether the threshold applies <strong>overall or to every task</strong>. Our default gate requires at least 95% observed accuracy overall and per task, with no critical violations. With three attempts per task, this requires 3/3: 2/3 falls below 95%. A configuration scoring 80/81 overall (98.8%) can therefore fail the per-task gate. An overall-only gate admits different configurations without changing their scores.</p>
<p>This is an observed selection threshold, not a confidence bound on reliability. With only three attempts per task, it operates as an all-attempts-must-pass screen. The gate uses task accuracy and critical violations; strict output-contract compliance is reported separately.</p>
<h2>Build a varied task set</h2>
<p>The benchmark contains <strong>27 distinct complex tasks across nine domains</strong>, three tasks per domain. Each provides a fictional source pack with the necessary facts and rules; the GPT-6 sweep (one complete evaluation run across the benchmark matrix) uses no external search or tools.</p>
<p>These domains test different reasoning skills: optimization under constraints, reconciling calculations with authoritative records, resource allocation, causal explanation, and safe sequencing. They let us compare how the same configuration handles different business and engineering workloads.</p>
<p>Strong early performance shaped the experiment. Every Astra setting solved every attempt in the earlier mixed-difficulty benchmark. I expanded the task set and increased its reasoning complexity by adding interacting constraints, multiple steps, and explicit optimality or safety requirements. Simpler cases remain as regression tests. As models improve, familiar questions may no longer reveal differences.</p>
<p>These tasks form a development benchmark used to inspect differences and build a shortlist. An untouched application validation set is needed before treating the selected configuration as a deployment choice.</p>
<table>
<thead>
<tr>
<th>Domain</th>
<th>What the tasks exercise</th>
</tr>
</thead>
<tbody><tr>
<td>Operations planning</td>
<td>Fulfillment optimization, production scheduling, warehouse queues</td>
</tr>
<tr>
<td>Procurement</td>
<td>Sourcing constraints, contingent purchases, invoice matching</td>
</tr>
<tr>
<td>Liquidity</td>
<td>Cash ledgers, currency netting, refinancing covenants</td>
</tr>
<tr>
<td>Financial diligence</td>
<td>EBITDA reconciliation, revenue cohorts, cash earnings</td>
</tr>
<tr>
<td>Transformation</td>
<td>Roadmap dependencies, impact estimates, automation policy</td>
</tr>
<tr>
<td>Engagement delivery</td>
<td>Skills and capacity, work assignments, rescue plans</td>
</tr>
<tr>
<td>Incident response</td>
<td>Safe retries, fault explanations, lease fencing</td>
</tr>
<tr>
<td>Disaster recovery</td>
<td>Backup chains, consistent restore points, resilient capacity</td>
</tr>
<tr>
<td>Release engineering</td>
<td>Canary analysis, migration sequences, version compatibility</td>
</tr>
</tbody></table>
<p>These are different task types, not numerical variants of a single template. Difficulty is a design judgment. Expected answers are frozen, checked against reference solvers and source data, and tested with valid and invalid grader inputs; experts have not reviewed them independently.</p>
<h3>Example: How much of this invoice should we pay?</h3>
<p>One task asks the model to reconcile an invoice with a purchase order, delivery records, returns, and credits. The records are summarized below; each line is evaluated separately.</p>
<table>
<thead>
<tr>
<th>Line</th>
<th>Order and agreed price</th>
<th>Received / returned</th>
<th>Invoice</th>
<th>Credit</th>
</tr>
</thead>
<tbody><tr>
<td>P1</td>
<td>10 X at $100 each</td>
<td>6 X + 8 Y received; 2 Y returned</td>
<td>10 X at $100</td>
<td>$50</td>
</tr>
<tr>
<td>P2</td>
<td>8 X at $120 each</td>
<td>8 X received; 1 X returned</td>
<td>8 X at $125</td>
<td>None</td>
</tr>
<tr>
<td>P3</td>
<td>5 Z at $80 each</td>
<td>4 Z received</td>
<td>5 Z at $80</td>
<td>None</td>
</tr>
</tbody></table>
<p>A signed agreement allows <strong>two Y units to replace one X</strong>. An unsigned draft proposes a one-for-one substitution. Receipt R2 (the eight Y units) and credit C1 ($50) each appear twice in the source pack.</p>
<p>The abbreviated prompt captures the decision rules:</p>
<pre><code class="language-text">Determine how much to pay on each invoice line.
Count duplicate receipt and credit IDs once; subtract returns.
Use the signed substitution ratio. Ignore the unsigned draft.
Do not move surplus quantities between purchase-order lines.

Quantity matched = minimum of net received, invoiced, and ordered
quantities, expressed in equivalent ordered units.
Held quantity = invoiced quantity minus quantity matched.
Held quantity measures shortages, even when the price is disputed.

If the invoice price differs from the agreed price, pay $0
for that entire line and report price_dispute.
Otherwise pay quantity matched × agreed price, subtract posted
credits once, and floor at $0. Report short_receipt or matched.

Return JSON with payable_dollars, held_equivalents, reasons,
total_payable_dollars, deduplicated_record_ids, and evidence_ids.
Sort ID lists lexicographically; cite authoritative evidence only.
</code></pre>
<p>The checked answer is <strong>$1,170, payable as follows</strong>: $850 for P1, $0 for the price-disputed P2, and $320 for P3. Getting there requires more than arithmetic: the model must deduplicate records, convert substitutions, distinguish shortages from price disputes, and follow the signed evidence. See the <a href="https://github.com/garystafford/openai-reasoning-benchmark/blob/main/benchmarks/reasoning/prompts.json">complete prompt and source pack</a> (<code>invoice_match</code>).</p>
<h2>Configure a fair comparison with Promptfoo</h2>
<p>The repository was converted to Promptfoo to bring execution, grading, and inspection into one workflow. Promptfoo runs the configuration matrix, applies assertions, and saves the responses and metrics.</p>
<p>A repository adapter delegates transport to Promptfoo’s native OpenAI Responses provider, supplies the task-specific schema, calculates usage-based cost, and selects the final assistant answer for grading:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/ab4a755b-4c6a-42e5-9c29-3231f3354147.png" alt="Evaluation workflow using Promptfoo and the Responses API" style="display:block;margin:0 auto" />

<p><em>Figure 2. Shared source packs and declared contracts flow through the same evaluation harness for every configuration.</em></p>
<p>Every request uses OpenAI <a href="https://developers.openai.com/api/docs/guides/structured-outputs">Structured Outputs</a>: <code>text.format</code> with JSON Schema and <code>strict: true</code>. Schemas define field types and input identifiers, without embedding solved answers.</p>
<p>Structured Outputs constrains completed, non-refusal answers to the supplied schema. It does not establish factual correctness, optimality, evidence support, or every task-specific ordering rule. Refusals and incomplete responses require separate handling.</p>
<p>A simplified request fragment illustrates the contract:</p>
<pre><code class="language-javascript">const requestFragment = {
  model: "gpt-6-sol",
  reasoning: { effort: "low" },
  text: {
    format: {
      type: "json_schema",
      name: "payment_summary",
      strict: true,
      schema: {
        type: "object",
        properties: { total_payable_dollars: { type: "integer" } },
        required: ["total_payable_dollars"],
        additionalProperties: false
      }
    }
  }
};
</code></pre>
<p>Each task has its own schema. Deterministic <a href="https://www.promptfoo.dev/docs/configuration/expected-outputs/python/">Python assertions</a> check arithmetic, feasibility, optimality, evidence, and unsupported conclusions.</p>
<p><strong>Task accuracy</strong> measures the decision; <strong>strict correctness</strong> also requires compliance with the output contract. Unordered selections compare as sets for accuracy, but must satisfy sorting rules for a strict pass. Execution sequences stay ordered. A <strong>critical violation</strong> breaks a hard constraint, such as a scheduling or recovery rule; a feasible but suboptimal plan can fail accuracy without one.</p>
<p><a href="https://developers.openai.com/api/docs/guides/reasoning">Reasoning effort</a> controls how much reasoning the model applies to a task. It is a model-dependent setting, not a fixed token budget or a guarantee that higher effort will produce a better answer.</p>
<p>The matrix uses 17 configurations: five Astra efforts (<code>low</code> through <code>max</code>) and six each for Sol and Luna (<code>none</code> through <code>max</code>). The model IDs are <code>gpt-6-astra</code>, <code>gpt-6-sol</code>, and <code>gpt-6-luna</code>. Check the <a href="https://developers.openai.com/api/docs/guides/latest-model">current model guidance</a> before adapting the comparison to another release. The three evaluation sweeps ran on October 1, 2026, using <code>gpt-6-astra</code>, <code>gpt-6-sol</code>, and <code>gpt-6-luna</code>; model behavior, available models, or pricing is subject to change.</p>
<p>The ignored env file supplies <code>OPENAI_API_KEY</code>. I ran three unchanged sweeps: <strong>27 tasks × 17 configurations × three attempts = 1,377 evaluated attempts</strong>. Requests use the Standard service tier, the default standard reasoning mode, default verbosity, and a concurrency of 4. Each request has a maximum output budget of 16,384 tokens, shared between reasoning tokens and the visible answer. The standard service tier and the standard reasoning mode are separate settings. The comparison measures each configuration under this shared output cap, not unconstrained reasoning performance. When adapting the harness, inspect incomplete statuses and token usage before attributing a failure to the model's capabilities. Promptfoo response caching is disabled; API input-prefix caching can still occur. Source hashes verify matching benchmark definitions across sweeps.</p>
<p>To run three attempts per task and configuration in one evaluation, set <code>BENCHMARK_RUNS=3</code>. The results presented here combine three separately executed sweeps.</p>
<p>Promptfoo’s headline passing percentage reflects strict grading. The article’s accuracy charts use <code>task_success</code>, so a sorting-only failure does not become a wrong business decision.</p>
<p>The report checks coverage, distinct uncached responses, comparable settings, and critical violations. Use <code>--accuracy-gate overall --min-accuracy 0.95</code> when your requirement is aggregate accuracy.</p>
<p>Latency is the elapsed time for the provider call, including any retries and service delays within that call. It excludes total matrix runtime. We report medians; applications with deadlines may also need p95 latency on a representative workload.</p>
<p><strong>Why repeat the same tasks?</strong> A configuration can answer correctly once and miss a requirement on its next attempt. Repeats expose variation and provide cost and latency summaries with a larger sample, without adding new benchmark questions.</p>
<h2>Compare accuracy, latency, and cost</h2>
<p>Across the sweeps, <strong>1,234 of 1,377 attempts passed task accuracy; 1,229 passed strict grading</strong>. Completed, non-refusal answers had no schema violations or final-answer extraction failures.</p>
<p>The known-usage estimated cost subtotal was <strong>$19.54</strong>, including failed answers. One Luna/none API refusal supplied no answer or usage. It counts as an unsuccessful, ungraded attempt with unknown cost; Luna/none is omitted from cost comparisons.</p>
<p>There were 29 critical violations: 28 in <code>none</code> configurations and one in Luna/low. Valid structure did not prevent substantive errors or the five strict-only failures.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/cbf2659b-93eb-4afb-a8fb-7eda9d653d59.png" alt="Accuracy for all 17 model and reasoning configurations" style="display:block;margin:0 auto" />

<p><em>Figure 3. Each configuration has 81 attempts on 27 distinct tasks. Bars count successful task attempts; markers count strict passes. The hatched segment identifies the ungraded API refusal.</em></p>
<p><strong>Astra/low is a strong default in this benchmark.</strong> It achieved 81/81 task successes and strict passes, with no critical violations. Higher Astra efforts also reached 81/81, at greater cost and latency. If Astra/low meets your response-time and budget requirements, these results give no accuracy-based reason to increase its effort.</p>
<p>Under the default per-task gate, <strong>Luna/max was the least expensive qualifying configuration; Sol/high had the lowest median latency</strong>.</p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Correct responses</th>
<th>Strict passes</th>
<th>Mean estimated cost/attempt</th>
<th>Median latency</th>
</tr>
</thead>
<tbody><tr>
<td>GPT-6 Luna / max</td>
<td>81/81</td>
<td>81/81</td>
<td>$0.001505</td>
<td>13.889 s</td>
</tr>
<tr>
<td>GPT-6 Sol / high</td>
<td>81/81</td>
<td>81/81</td>
<td>$0.008646</td>
<td>5.015 s</td>
</tr>
<tr>
<td>GPT-6 Astra / low</td>
<td>81/81</td>
<td>81/81</td>
<td>$0.024565</td>
<td>5.265 s</td>
</tr>
<tr>
<td>GPT-6 Sol / low</td>
<td>78/81</td>
<td>77/81</td>
<td>$0.005723</td>
<td>3.722 s</td>
</tr>
</tbody></table>
<p>All four had zero observed critical violations. The first three meet the default per-task gate. Sol/low meets the overall-only 95% gate but misses the per-task requirement.</p>
<p>Compared with Astra/low, <strong>Sol/high reduced mean estimated cost per attempt by about 65%</strong>, with all 81 strict passes. Its median latency was slightly lower: 5.015 versus 5.265 seconds. The small latency difference does not establish a consistent speed advantage.</p>
<p><strong>Luna/max reduced mean estimated cost per attempt by about 94% relative to Astra/low</strong>, also with all 81 strict passes, but took about 2.6 times as long at the median. That trade-off may suit a background job with a flexible deadline.</p>
<p>Repeats changed the shortlist. <strong>Sol/low scored 27/27, 26/27, and 25/27.</strong> Its later regional-failover plans had modeled costs of $290 and $350 against a $260 optimum, and it returned a suboptimal fulfillment plan. The first sweep alone missed those errors.</p>
<p>Sol/low reduced mean estimated cost by about 77% and median latency by about 29% relative to Astra/low. Whether those savings justify its observed misses depends on the accuracy requirement.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/a485a96f-dbf6-4ec3-b8e3-7fd3adfaee24.png" alt="Accuracy plotted against estimated cost by model" style="display:block;margin:0 auto" />

<p><em>Figure 4. Outlined points mark each model’s least expensive qualifying effort. Panels use separate linear cost scales; compare dollar labels. Luna/none’s cost is unavailable.</em></p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/df0646fa-6a2d-4bfa-b706-db6a0ffbc528.png" alt="Accuracy plotted against median latency by model" style="display:block;margin:0 auto" />

<p><em>Figure 5. Outlined points mark each model’s fastest qualifying effort. Latency panels use separate scales.</em></p>
<p>In the combined view, higher points indicate greater accuracy, points farther left indicate greater speed, and smaller bubbles indicate lower cost.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/1571649a-8107-432d-80c4-881f3b18332b.png" alt="Accuracy and latency with bubble area representing mean estimated cost" style="display:block;margin:0 auto" />

<p><em>Figure 6. The bubble area represents the mean estimated cost per attempt; all models share the same accuracy and latency scales. Luna/none is omitted because its cost is incomplete. The vertical scale begins at 40%.</em></p>
<p>With an <strong>overall threshold of only 95%</strong>, Luna/high becomes the least expensive qualifying choice: 77/81 (95.1%), $0.000774 per attempt, and a median of 7.979 seconds. Sol/low becomes the fastest. Both fail the per-task gate. These are different selection rules applied to the same results.</p>
<p>Costs are based on recorded usage and the repository’s frozen September 28 rate card. API input-prefix caching differed across sweeps and affected costs; disabling response caching does not disable prefix caching. These estimates are not billed totals. The cost rankings include the caching conditions of these sweeps, rather than a standardized cold-cache workload. The estimator separates ordinary input, cache reads, cache writes, and output; absent cache-detail fields are treated as zero, so preserve the raw usage and verify those fields before claiming complete accounting. See the <a href="https://developers.openai.com/api/docs/guides/prompt-caching">prompt caching guide</a> for the separate cache-read and cache-write charges. Preserve usage and rates with the results, and check <a href="https://developers.openai.com/api/docs/pricing">current pricing</a> for your workload.</p>
<h2>A small GPT-5.6 comparison</h2>
<p><strong>GPT-6 Sol/high matched GPT-5.6 Sol/high’s accuracy, while costing 59% less and delivering 30% lower median latency.</strong> Both passed all 27 tasks. The mean estimated cost per task fell from $0.02331 to $0.00953, and the median latency fell from 7.475 to 5.264 seconds. GPT-5.6 Sol/low scored 25/27, missing a fulfillment optimum and an incident attempt count.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/0954ae7f-831c-4936-bbb6-0b94b1569bf2.png" alt="GPT-5.6 and GPT-6 Sol compared on accuracy, latency, and cost" style="display:block;margin:0 auto" />

<p><em>Figure 7. One attempt on the same 27 tasks, using identical prompts, schemas, graders, and run settings. GPT-6 values use the first sweep, keeping this comparison separate from the three-attempt results.</em></p>
<p>These observations support testing a newer model against your existing workload. One attempt per task does not ensure reliability; separate runtimes and input-prefix caching can affect latency and cost.</p>
<h2>Match the configuration to the domain</h2>
<p>The overall score hides useful differences. Luna/low passed 67 of 81 attempts, yet stayed at 9/9 in transformation, engagement delivery, disaster recovery, and release engineering. For those domain samples, it was the least expensive qualifying choice.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/e1d73fd8-8a54-4069-abbb-ebe780841a83.png" alt="Task accuracy by domain and configuration" style="display:block;margin:0 auto" />

<p><em>Figure 8. Every cell shows the correct responses across nine attempts: three distinct tasks, three attempts each. A 9/9 cell is an observation in this sample, not a domain reliability estimate.</em></p>
<p>Sol/low was the fastest among qualifying configurations across seven domain samples; Astra/low led in operations planning, and Sol/medium in disaster recovery. Repeats changed the procurement cost choice from Luna/medium in the first sweep to Luna/max across all three.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9c3e809d-0ffd-4a99-84af-a95450bef0e2.png" alt="Least expensive and fastest qualifying choices by domain" style="display:block;margin:0 auto" />

<p><em>Figure 9. Mean estimated cost and median latency for the selected domain configurations. Each selection requires all nine responses to succeed, with no critical violations observed.</em></p>
<p>The qualifying choices differ by domain under the same selection rule. Use them as a shortlist for your own tasks.</p>
<h2>Inspect failures that change the decision</h2>
<p>Structured Outputs enforces structure; the grader still has to measure the right decision. The following examples distinguish decision errors, output-contract failures, ambiguous requirements, and final-answer selection.</p>
<p><strong>Correct plan, wrong ordering.</strong> Sol/low returned the optimal $1,090 fulfillment plan. These fields are excerpted from its answer:</p>
<pre><code class="language-json">{
  "selected_orders": ["O1", "O3", "O5", "O6", "O8", "O10"],
  "net_value_dollars": 1090
}
</code></pre>
<p>Alphabetical ordering means lexicographic string order: O10 precedes O3. The schema validates array structure but does not impose ordering. The grader’s named scores distinguish the correct decision from the strict failure:</p>
<pre><code class="language-json">{
  "task_success": 1,
  "format_compliance": 0,
  "critical_violation": 0,
  "strict_correctness": 0
}
</code></pre>
<p><strong>Feasible plan, lower value.</strong> Luna/medium returned a feasible $1,080 fulfillment plan against the $1,090 optimum. It passed format compliance and had no critical violations, but failed task accuracy because the task requires the optimum. An application could instead define an acceptable dollar shortfall before running the evaluation.</p>
<p><strong>Ambiguous business meaning.</strong> In the invoice task, a purchase-order line has eight invoiced equivalents and seven quantity-matched equivalents. Its invoice price also differs from the signed price. The required output includes:</p>
<pre><code class="language-json">{
  "held_equivalents": { "P2": 1 },
  "payable_dollars": { "P2": 0 },
  "reasons": { "P2": "price_dispute" }
}
</code></pre>
<p>The held equivalent is the quantity shortfall; the price dispute separately blocks payment. We clarified that distinction before these sweeps, without changing the expected answer. All 51 responses returned the intended held quantity and duplicate-ID list; 48 passed the full task.</p>
<p><strong>Selecting the completed answer.</strong> OpenAI’s <a href="https://developers.openai.com/api/docs/guides/reasoning">message-phase guidance</a> distinguishes commentary from <code>final_answer</code>. Luna/none returned both on the fault-hypothesis task. The adapter preserved the raw messages and graded only the final answer, which passed.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/05e2d80b-ce3a-44b7-be60-af4e959dc4c8.png" alt="Task correctness and strict-only failures across the sweep" style="display:block;margin:0 auto" />

<p><em>Figure 10. The 1,377 attempts’ separate strict passes, strict-only failures, substantive mistakes, and an ungraded API refusal. Critical violations are tracked separately.</em></p>
<h2>Apply the method to your workload</h2>
<p>Start with representative application tasks and agreed scoring rules. Check the grader with answers that should pass and those that should fail. Compare configurations using the same prompts and output contracts, repeat the attempts, and inspect the misses before choosing.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/9398440c-6500-42ae-9069-c49d79a6e86e.png" alt="Select a configuration using accuracy, latency, and cost requirements" style="display:block;margin:0 auto" />

<p><em>Figure 11. Apply requirements first, then compare the eligible configurations.</em></p>
<p><strong>A perfect run doesn’t establish reliability.</strong> These results describe performance on this benchmark, not expected production performance. Fictional source packs make the evidence and expected answers reproducible, but three attempts per task cannot establish reliability, and three tasks cannot represent an entire domain. Real-world data, noisy inputs, retrieval, tool use, and longer conversations could affect the accuracy rankings and the observed differences in cost and latency. Validate your shortlist against a larger, representative application workload that includes failure cases and tail latency, while preserving the task definitions, scoring rules, model IDs, and run conditions.</p>
<h2>Inspect and reproduce the comparison</h2>
<p>Each viewer row is a task attempt; each column is a model/reasoning configuration. Headers summarize scores, tokens, cost, and average latency. Use <strong>Columns</strong> to focus a comparison, <strong>Failures</strong> to filter strict failures, and cell details to inspect prompts, outputs, and grading.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/207b1543-cc7f-45f0-8885-71bba3fbcbcc.jpg" alt="Promptfoo results interface with two model configurations selected" style="display:block;margin:0 auto" />

<p><em>Figure 12. The combined view has 17 providers and 81 task-attempt rows. The selected columns compare Sol/low and Luna/max; the green percentage shows the strict pass rate. Column selection changes the view, not the dataset.</em></p>
<p>Click <strong>Show Charts</strong> to view summaries of strict pass rates and named scores before inspecting individual answers.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/d132c4e0-1eeb-4d84-8284-e7abfe396e49.jpg" alt="Promptfoo Show Charts view with native pass-rate and named-score charts" style="display:block;margin:0 auto" />

<p><em>Figure 13. Promptfoo’s native charts summarize the three-sweep results. The publication charts earlier in this article use median latency; the viewer displays average latency.</em></p>
<p><strong>Eval actions → View YAML</strong> shows the run’s saved provider settings:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/bf0549a5-b78e-4085-87f6-24695492bef8.jpg" alt="Saved Promptfoo configuration showing the Astra low provider settings" style="display:block;margin:0 auto" />

<p><em>Figure 14. The Astra/low provider entry shows the API endpoint, service tier, and output-token budget.</em></p>
<p>Preview the scope, run one attempt per task/configuration, then inspect and report the saved results:</p>
<pre><code class="language-shell">npm ci
npm test
# Preview without API calls
BENCHMARK_RUNS=1 BENCHMARK_CONCURRENCY=4 npm run eval:matrix:plan
# Run 459 task/configuration pairs
BENCHMARK_RUNS=1 BENCHMARK_CONCURRENCY=4 \
  npm run eval:matrix -- --env-file .env.local
npm run eval:view -- --port 15500
npm run eval:report -- results/promptfoo-YOUR-RUN.json
</code></pre>
<p>The commands above run one sweep. Set <code>BENCHMARK_RUNS=3</code> for three attempts per task and configuration in one evaluation. The published results instead combine three separate sweeps. Raw evaluation exports and publication-figure scripts are not included in the public distribution.</p>
<h2>A brief GPT-6.1 agent experiment</h2>
<p>I also tried GPT-6.1 Sol at medium reasoning on diligence reconciliation, with hosted multi-agent support enabled. The first call produced no subagents. A second call explicitly requested three independent checks: closing entries and EBITDA; addbacks, debt, leverage, and valuation; and source authority and missing evidence.</p>
<p>Note: At the time of this experiment, hosted multi-agent support is a beta feature requiring explicit API opt-in. See the <a href="https://developers.openai.com/api/docs/guides/responses-multi-agent">Responses multi-agent documentation</a> for supported models and setup requirements.</p>
<p>The second call spawned <code>closing_check</code>, <code>financial_check</code>, and <code>authority_check</code>. The response exposes their returned names and recorded collaboration actions. Inter-agent messages are <a href="https://developers.openai.com/api/docs/guides/responses-multi-agent">encrypted</a>, so the diagram shows requested scopes, not their conversations.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/29292c1b-fedd-46fd-b478-b168140613fc.png" alt="Observed GPT-6.1 delegation with requested independent-check scopes" style="display:block;margin:0 auto" />

<p><em>Figure 15. Three subagents observed after explicit delegation. Enabling the capability alone did not produce any subagents in the other call.</em></p>
<p>Both final answers passed the same task grader. The explicitly delegated call took longer and had a higher estimated cost:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6abe4a89fbf68ad459d66120/5edf5262-39f6-49dc-9df0-434c07af0a7a.png" alt="Two exploratory GPT-6.1 diligence calls" style="display:block;margin:0 auto" />

<p><em>Figure 16. Two observations on a single task, under different prompts and caching conditions. This is not a controlled estimate of delegation overhead.</em></p>
<p>The follow-up post, <a href="https://blog.latentthoughts.com/beyond-a-single-agent-subagent-delegation-with-gpt-6-1-sol">Beyond a Single Agent: Subagent Delegation with GPT-6.1 Sol</a>, compares delegation across 12 tasks, with matched repetitions and separate measurements of accuracy, time, and estimated cost.</p>
<h2>Source code</h2>
<p>The <a href="https://github.com/garystafford/openai-reasoning-benchmark">repository</a> contains the Promptfoo configuration, source packs, output contracts, answer keys, reference solvers, deterministic graders, and report generator, so you can inspect the evaluation logic and run the benchmark on your own configurations. The raw evaluation exports and scripts used to combine sweeps and generate the publication figures are not included.</p>
<p><a class="embed-card" href="https://github.com/garystafford/openai-reasoning-benchmark">https://github.com/garystafford/openai-reasoning-benchmark</a></p>

<hr />
<p><em>This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.</em></p>
]]></content:encoded></item></channel></rss>