Skip to main content

Choosing the Right OpenAI Model and Reasoning Level

An evaluation guide to accuracy, latency, and cost across complex workloads.

Updated 20 min readMarkdown
Choosing the Right OpenAI Model and Reasoning Level

Your AI application needs answers it can trust, delivered fast enough and at a cost that makes sense. Choosing the right model and reasoning level means defining those requirements and testing which configurations meet them.

In my earlier post, I compared models across multiple providers. This post compares GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna on 27 constructed reasoning tasks, using three evaluation sweeps run on October 1, 2026. The main comparison uses gpt-6-sol, not the newer gpt-6.1-sol; a separate exploratory example uses GPT-6.1 Sol.

Across these tasks, Luna/max was the least expensive configuration meeting the default selection gate, while Sol/high had the lowest observed median latency among qualifying configurations. Repeated attempts exposed errors that the first sweep missed, changing which configurations qualified. These are benchmark observations, not production reliability estimates.

Define what success means

Evaluate against the requirements of your application:

Reason to evaluate Question to answer
Accuracy Does it get your tasks right?
Latency Is it fast enough for the application?
Cost Is the quality worth the cost per task?
Consistency Does it succeed across repeated attempts?
Workload fit Which model and reasoning level works best for each use case?

This post focuses on accuracy, latency, and cost, using repeated attempts and different domains to examine consistency and workload fit.

An interactive application may need a quick answer; a background planning job may take longer to produce a better plan. A small error may be acceptable in an estimate and unacceptable in a release gate.

A key part of defining requirements is deciding what counts as correct and making sure the grader measures it fairly.

Does a correct schedule need to be feasible or optimal? What units and rounding rules apply? Does list order matter? Define these requirements before looking at model scores.

Define accuracy, latency, and cost requirements

Figure 1. Keep the requirements short enough to use and precise enough to grade.

Specify whether the threshold applies overall or to every task. Our default gate requires at least 95% observed accuracy overall and per task, with no critical violations. With three attempts per task, this requires 3/3: 2/3 falls below 95%. A configuration scoring 80/81 overall (98.8%) can therefore fail the per-task gate. An overall-only gate admits different configurations without changing their scores.

This is an observed selection threshold, not a confidence bound on reliability. With only three attempts per task, it operates as an all-attempts-must-pass screen. The gate uses task accuracy and critical violations; strict output-contract compliance is reported separately.

Build a varied task set

The benchmark contains 27 distinct complex tasks across nine domains, three tasks per domain. Each provides a fictional source pack with the necessary facts and rules; the GPT-6 sweep (one complete evaluation run across the benchmark matrix) uses no external search or tools.

These domains test different reasoning skills: optimization under constraints, reconciling calculations with authoritative records, resource allocation, causal explanation, and safe sequencing. They let us compare how the same configuration handles different business and engineering workloads.

Strong early performance shaped the experiment. Every Astra setting solved every attempt in the earlier mixed-difficulty benchmark. I expanded the task set and increased its reasoning complexity by adding interacting constraints, multiple steps, and explicit optimality or safety requirements. Simpler cases remain as regression tests. As models improve, familiar questions may no longer reveal differences.

These tasks form a development benchmark used to inspect differences and build a shortlist. An untouched application validation set is needed before treating the selected configuration as a deployment choice.

Domain What the tasks exercise
Operations planning Fulfillment optimization, production scheduling, warehouse queues
Procurement Sourcing constraints, contingent purchases, invoice matching
Liquidity Cash ledgers, currency netting, refinancing covenants
Financial diligence EBITDA reconciliation, revenue cohorts, cash earnings
Transformation Roadmap dependencies, impact estimates, automation policy
Engagement delivery Skills and capacity, work assignments, rescue plans
Incident response Safe retries, fault explanations, lease fencing
Disaster recovery Backup chains, consistent restore points, resilient capacity
Release engineering Canary analysis, migration sequences, version compatibility

These are different task types, not numerical variants of a single template. Difficulty is a design judgment. Expected answers are frozen, checked against reference solvers and source data, and tested with valid and invalid grader inputs; experts have not reviewed them independently.

Example: How much of this invoice should we pay?

One task asks the model to reconcile an invoice with a purchase order, delivery records, returns, and credits. The records are summarized below; each line is evaluated separately.

Line Order and agreed price Received / returned Invoice Credit
P1 10 X at $100 each 6 X + 8 Y received; 2 Y returned 10 X at $100 $50
P2 8 X at $120 each 8 X received; 1 X returned 8 X at $125 None
P3 5 Z at $80 each 4 Z received 5 Z at $80 None

A signed agreement allows two Y units to replace one X. An unsigned draft proposes a one-for-one substitution. Receipt R2 (the eight Y units) and credit C1 ($50) each appear twice in the source pack.

The abbreviated prompt captures the decision rules:

Determine how much to pay on each invoice line.
Count duplicate receipt and credit IDs once; subtract returns.
Use the signed substitution ratio. Ignore the unsigned draft.
Do not move surplus quantities between purchase-order lines.

Quantity matched = minimum of net received, invoiced, and ordered
quantities, expressed in equivalent ordered units.
Held quantity = invoiced quantity minus quantity matched.
Held quantity measures shortages, even when the price is disputed.

If the invoice price differs from the agreed price, pay $0
for that entire line and report price_dispute.
Otherwise pay quantity matched × agreed price, subtract posted
credits once, and floor at $0. Report short_receipt or matched.

Return JSON with payable_dollars, held_equivalents, reasons,
total_payable_dollars, deduplicated_record_ids, and evidence_ids.
Sort ID lists lexicographically; cite authoritative evidence only.

The checked answer is $1,170, payable as follows: $850 for P1, $0 for the price-disputed P2, and $320 for P3. Getting there requires more than arithmetic: the model must deduplicate records, convert substitutions, distinguish shortages from price disputes, and follow the signed evidence. See the complete prompt and source pack (invoice_match).

Configure a fair comparison with Promptfoo

The repository was converted to Promptfoo to bring execution, grading, and inspection into one workflow. Promptfoo runs the configuration matrix, applies assertions, and saves the responses and metrics.

A repository adapter delegates transport to Promptfoo’s native OpenAI Responses provider, supplies the task-specific schema, calculates usage-based cost, and selects the final assistant answer for grading:

Evaluation workflow using Promptfoo and the Responses API

Figure 2. Shared source packs and declared contracts flow through the same evaluation harness for every configuration.

Every request uses OpenAI Structured Outputs: text.format with JSON Schema and strict: true. Schemas define field types and input identifiers, without embedding solved answers.

Structured Outputs constrains completed, non-refusal answers to the supplied schema. It does not establish factual correctness, optimality, evidence support, or every task-specific ordering rule. Refusals and incomplete responses require separate handling.

A simplified request fragment illustrates the contract:

const requestFragment = {
  model: "gpt-6-sol",
  reasoning: { effort: "low" },
  text: {
    format: {
      type: "json_schema",
      name: "payment_summary",
      strict: true,
      schema: {
        type: "object",
        properties: { total_payable_dollars: { type: "integer" } },
        required: ["total_payable_dollars"],
        additionalProperties: false
      }
    }
  }
};

Each task has its own schema. Deterministic Python assertions check arithmetic, feasibility, optimality, evidence, and unsupported conclusions.

Task accuracy measures the decision; strict correctness also requires compliance with the output contract. Unordered selections compare as sets for accuracy, but must satisfy sorting rules for a strict pass. Execution sequences stay ordered. A critical violation breaks a hard constraint, such as a scheduling or recovery rule; a feasible but suboptimal plan can fail accuracy without one.

Reasoning effort controls how much reasoning the model applies to a task. It is a model-dependent setting, not a fixed token budget or a guarantee that higher effort will produce a better answer.

The matrix uses 17 configurations: five Astra efforts (low through max) and six each for Sol and Luna (none through max). The model IDs are gpt-6-astra, gpt-6-sol, and gpt-6-luna. Check the current model guidance before adapting the comparison to another release. The three evaluation sweeps ran on October 1, 2026, using gpt-6-astra, gpt-6-sol, and gpt-6-luna; model behavior, available models, or pricing is subject to change.

The ignored env file supplies OPENAI_API_KEY. I ran three unchanged sweeps: 27 tasks × 17 configurations × three attempts = 1,377 evaluated attempts. Requests use the Standard service tier, the default standard reasoning mode, default verbosity, and a concurrency of 4. Each request has a maximum output budget of 16,384 tokens, shared between reasoning tokens and the visible answer. The standard service tier and the standard reasoning mode are separate settings. The comparison measures each configuration under this shared output cap, not unconstrained reasoning performance. When adapting the harness, inspect incomplete statuses and token usage before attributing a failure to the model's capabilities. Promptfoo response caching is disabled; API input-prefix caching can still occur. Source hashes verify matching benchmark definitions across sweeps.

To run three attempts per task and configuration in one evaluation, set BENCHMARK_RUNS=3. The results presented here combine three separately executed sweeps.

Promptfoo’s headline passing percentage reflects strict grading. The article’s accuracy charts use task_success, so a sorting-only failure does not become a wrong business decision.

The report checks coverage, distinct uncached responses, comparable settings, and critical violations. Use --accuracy-gate overall --min-accuracy 0.95 when your requirement is aggregate accuracy.

Latency is the elapsed time for the provider call, including any retries and service delays within that call. It excludes total matrix runtime. We report medians; applications with deadlines may also need p95 latency on a representative workload.

Why repeat the same tasks? A configuration can answer correctly once and miss a requirement on its next attempt. Repeats expose variation and provide cost and latency summaries with a larger sample, without adding new benchmark questions.

Compare accuracy, latency, and cost

Across the sweeps, 1,234 of 1,377 attempts passed task accuracy; 1,229 passed strict grading. Completed, non-refusal answers had no schema violations or final-answer extraction failures.

The known-usage estimated cost subtotal was $19.54, including failed answers. One Luna/none API refusal supplied no answer or usage. It counts as an unsuccessful, ungraded attempt with unknown cost; Luna/none is omitted from cost comparisons.

There were 29 critical violations: 28 in none configurations and one in Luna/low. Valid structure did not prevent substantive errors or the five strict-only failures.

Accuracy for all 17 model and reasoning configurations

Figure 3. Each configuration has 81 attempts on 27 distinct tasks. Bars count successful task attempts; markers count strict passes. The hatched segment identifies the ungraded API refusal.

Astra/low is a strong default in this benchmark. It achieved 81/81 task successes and strict passes, with no critical violations. Higher Astra efforts also reached 81/81, at greater cost and latency. If Astra/low meets your response-time and budget requirements, these results give no accuracy-based reason to increase its effort.

Under the default per-task gate, Luna/max was the least expensive qualifying configuration; Sol/high had the lowest median latency.

Configuration Correct responses Strict passes Mean estimated cost/attempt Median latency
GPT-6 Luna / max 81/81 81/81 $0.001505 13.889 s
GPT-6 Sol / high 81/81 81/81 $0.008646 5.015 s
GPT-6 Astra / low 81/81 81/81 $0.024565 5.265 s
GPT-6 Sol / low 78/81 77/81 $0.005723 3.722 s

All four had zero observed critical violations. The first three meet the default per-task gate. Sol/low meets the overall-only 95% gate but misses the per-task requirement.

Compared with Astra/low, Sol/high reduced mean estimated cost per attempt by about 65%, with all 81 strict passes. Its median latency was slightly lower: 5.015 versus 5.265 seconds. The small latency difference does not establish a consistent speed advantage.

Luna/max reduced mean estimated cost per attempt by about 94% relative to Astra/low, also with all 81 strict passes, but took about 2.6 times as long at the median. That trade-off may suit a background job with a flexible deadline.

Repeats changed the shortlist. Sol/low scored 27/27, 26/27, and 25/27. Its later regional-failover plans had modeled costs of $290 and $350 against a $260 optimum, and it returned a suboptimal fulfillment plan. The first sweep alone missed those errors.

Sol/low reduced mean estimated cost by about 77% and median latency by about 29% relative to Astra/low. Whether those savings justify its observed misses depends on the accuracy requirement.

Accuracy plotted against estimated cost by model

Figure 4. Outlined points mark each model’s least expensive qualifying effort. Panels use separate linear cost scales; compare dollar labels. Luna/none’s cost is unavailable.

Accuracy plotted against median latency by model

Figure 5. Outlined points mark each model’s fastest qualifying effort. Latency panels use separate scales.

In the combined view, higher points indicate greater accuracy, points farther left indicate greater speed, and smaller bubbles indicate lower cost.

Accuracy and latency with bubble area representing mean estimated cost

Figure 6. The bubble area represents the mean estimated cost per attempt; all models share the same accuracy and latency scales. Luna/none is omitted because its cost is incomplete. The vertical scale begins at 40%.

With an overall threshold of only 95%, Luna/high becomes the least expensive qualifying choice: 77/81 (95.1%), $0.000774 per attempt, and a median of 7.979 seconds. Sol/low becomes the fastest. Both fail the per-task gate. These are different selection rules applied to the same results.

Costs are based on recorded usage and the repository’s frozen September 28 rate card. API input-prefix caching differed across sweeps and affected costs; disabling response caching does not disable prefix caching. These estimates are not billed totals. The cost rankings include the caching conditions of these sweeps, rather than a standardized cold-cache workload. The estimator separates ordinary input, cache reads, cache writes, and output; absent cache-detail fields are treated as zero, so preserve the raw usage and verify those fields before claiming complete accounting. See the prompt caching guide for the separate cache-read and cache-write charges. Preserve usage and rates with the results, and check current pricing for your workload.

A small GPT-5.6 comparison

GPT-6 Sol/high matched GPT-5.6 Sol/high’s accuracy, while costing 59% less and delivering 30% lower median latency. Both passed all 27 tasks. The mean estimated cost per task fell from $0.02331 to $0.00953, and the median latency fell from 7.475 to 5.264 seconds. GPT-5.6 Sol/low scored 25/27, missing a fulfillment optimum and an incident attempt count.

GPT-5.6 and GPT-6 Sol compared on accuracy, latency, and cost

Figure 7. One attempt on the same 27 tasks, using identical prompts, schemas, graders, and run settings. GPT-6 values use the first sweep, keeping this comparison separate from the three-attempt results.

These observations support testing a newer model against your existing workload. One attempt per task does not ensure reliability; separate runtimes and input-prefix caching can affect latency and cost.

Match the configuration to the domain

The overall score hides useful differences. Luna/low passed 67 of 81 attempts, yet stayed at 9/9 in transformation, engagement delivery, disaster recovery, and release engineering. For those domain samples, it was the least expensive qualifying choice.

Task accuracy by domain and configuration

Figure 8. Every cell shows the correct responses across nine attempts: three distinct tasks, three attempts each. A 9/9 cell is an observation in this sample, not a domain reliability estimate.

Sol/low was the fastest among qualifying configurations across seven domain samples; Astra/low led in operations planning, and Sol/medium in disaster recovery. Repeats changed the procurement cost choice from Luna/medium in the first sweep to Luna/max across all three.

Least expensive and fastest qualifying choices by domain

Figure 9. Mean estimated cost and median latency for the selected domain configurations. Each selection requires all nine responses to succeed, with no critical violations observed.

The qualifying choices differ by domain under the same selection rule. Use them as a shortlist for your own tasks.

Inspect failures that change the decision

Structured Outputs enforces structure; the grader still has to measure the right decision. The following examples distinguish decision errors, output-contract failures, ambiguous requirements, and final-answer selection.

Correct plan, wrong ordering. Sol/low returned the optimal $1,090 fulfillment plan. These fields are excerpted from its answer:

{
  "selected_orders": ["O1", "O3", "O5", "O6", "O8", "O10"],
  "net_value_dollars": 1090
}

Alphabetical ordering means lexicographic string order: O10 precedes O3. The schema validates array structure but does not impose ordering. The grader’s named scores distinguish the correct decision from the strict failure:

{
  "task_success": 1,
  "format_compliance": 0,
  "critical_violation": 0,
  "strict_correctness": 0
}

Feasible plan, lower value. Luna/medium returned a feasible $1,080 fulfillment plan against the $1,090 optimum. It passed format compliance and had no critical violations, but failed task accuracy because the task requires the optimum. An application could instead define an acceptable dollar shortfall before running the evaluation.

Ambiguous business meaning. In the invoice task, a purchase-order line has eight invoiced equivalents and seven quantity-matched equivalents. Its invoice price also differs from the signed price. The required output includes:

{
  "held_equivalents": { "P2": 1 },
  "payable_dollars": { "P2": 0 },
  "reasons": { "P2": "price_dispute" }
}

The held equivalent is the quantity shortfall; the price dispute separately blocks payment. We clarified that distinction before these sweeps, without changing the expected answer. All 51 responses returned the intended held quantity and duplicate-ID list; 48 passed the full task.

Selecting the completed answer. OpenAI’s message-phase guidance distinguishes commentary from final_answer. Luna/none returned both on the fault-hypothesis task. The adapter preserved the raw messages and graded only the final answer, which passed.

Task correctness and strict-only failures across the sweep

Figure 10. The 1,377 attempts’ separate strict passes, strict-only failures, substantive mistakes, and an ungraded API refusal. Critical violations are tracked separately.

Apply the method to your workload

Start with representative application tasks and agreed scoring rules. Check the grader with answers that should pass and those that should fail. Compare configurations using the same prompts and output contracts, repeat the attempts, and inspect the misses before choosing.

Select a configuration using accuracy, latency, and cost requirements

Figure 11. Apply requirements first, then compare the eligible configurations.

A perfect run doesn’t establish reliability. These results describe performance on this benchmark, not expected production performance. Fictional source packs make the evidence and expected answers reproducible, but three attempts per task cannot establish reliability, and three tasks cannot represent an entire domain. Real-world data, noisy inputs, retrieval, tool use, and longer conversations could affect the accuracy rankings and the observed differences in cost and latency. Validate your shortlist against a larger, representative application workload that includes failure cases and tail latency, while preserving the task definitions, scoring rules, model IDs, and run conditions.

Inspect and reproduce the comparison

Each viewer row is a task attempt; each column is a model/reasoning configuration. Headers summarize scores, tokens, cost, and average latency. Use Columns to focus a comparison, Failures to filter strict failures, and cell details to inspect prompts, outputs, and grading.

Promptfoo results interface with two model configurations selected

Figure 12. The combined view has 17 providers and 81 task-attempt rows. The selected columns compare Sol/low and Luna/max; the green percentage shows the strict pass rate. Column selection changes the view, not the dataset.

Click Show Charts to view summaries of strict pass rates and named scores before inspecting individual answers.

Promptfoo Show Charts view with native pass-rate and named-score charts

Figure 13. Promptfoo’s native charts summarize the three-sweep results. The publication charts earlier in this article use median latency; the viewer displays average latency.

Eval actions → View YAML shows the run’s saved provider settings:

Saved Promptfoo configuration showing the Astra low provider settings

Figure 14. The Astra/low provider entry shows the API endpoint, service tier, and output-token budget.

Preview the scope, run one attempt per task/configuration, then inspect and report the saved results:

npm ci
npm test
# Preview without API calls
BENCHMARK_RUNS=1 BENCHMARK_CONCURRENCY=4 npm run eval:matrix:plan
# Run 459 task/configuration pairs
BENCHMARK_RUNS=1 BENCHMARK_CONCURRENCY=4 \
  npm run eval:matrix -- --env-file .env.local
npm run eval:view -- --port 15500
npm run eval:report -- results/promptfoo-YOUR-RUN.json

The commands above run one sweep. Set BENCHMARK_RUNS=3 for three attempts per task and configuration in one evaluation. The published results instead combine three separate sweeps. Raw evaluation exports and publication-figure scripts are not included in the public distribution.

A brief GPT-6.1 agent experiment

I also tried GPT-6.1 Sol at medium reasoning on diligence reconciliation, with hosted multi-agent support enabled. The first call produced no subagents. A second call explicitly requested three independent checks: closing entries and EBITDA; addbacks, debt, leverage, and valuation; and source authority and missing evidence.

Note: At the time of this experiment, hosted multi-agent support is a beta feature requiring explicit API opt-in. See the Responses multi-agent documentation for supported models and setup requirements.

The second call spawned closing_check, financial_check, and authority_check. The response exposes their returned names and recorded collaboration actions. Inter-agent messages are encrypted, so the diagram shows requested scopes, not their conversations.

Observed GPT-6.1 delegation with requested independent-check scopes

Figure 15. Three subagents observed after explicit delegation. Enabling the capability alone did not produce any subagents in the other call.

Both final answers passed the same task grader. The explicitly delegated call took longer and had a higher estimated cost:

Two exploratory GPT-6.1 diligence calls

Figure 16. Two observations on a single task, under different prompts and caching conditions. This is not a controlled estimate of delegation overhead.

The follow-up post, Beyond a Single Agent: Subagent Delegation with GPT-6.1 Sol, compares delegation across 12 tasks, with matched repetitions and separate measurements of accuracy, time, and estimated cost.

Source code

The repository contains the Promptfoo configuration, source packs, output contracts, answer keys, reference solvers, deterministic graders, and report generator, so you can inspect the evaluation logic and run the benchmark on your own configurations. The raw evaluation exports and scripts used to combine sweeps and generate the publication figures are not included.

https://github.com/garystafford/openai-reasoning-benchmark


This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.

About the author

  • Applied AI Engineer at OpenAI. I write about building AI applications, evaluating models, and turning experiments into practical solutions, drawing on my background in cloud architecture and software engineering.

The repeat runs are the part that stood out to me. A model can look solid on one pass, then miss the same kind of task later.

Thanks, Julian. GPT-6 Sol at low reasoning got all 27 tasks right on the first pass, then scored 26/27 and 25/27 on the same tasks. On the regional failover task, the two later runs proposed plans costing $290 and $350 against a $260 optimum. The first pass would have put Sol/low on the shortlist. The repeats showed it missed our per-task requirement even though its overall accuracy stayed above 95%.

All I want to know is how these models are helping the enterprises... because from where I am seeing it from, the adoption in enterprise is still in phases and there is a huge deployment gap.

The use cases enterprises are building (or trying to build) can currently be done easily with the previous-generation models.

I agree with your observations, especially on the deployment gap. Many enterprise use cases can be served well by previous-generation models. The question is whether a newer model delivers a measurable improvement that justifies adopting it.

This demonstration uses one newer model family and a specific mix of hypothetical complex tasks, but the evaluation approach applies across use cases, domains, and model generations. The goal is to choose what meets your requirements, not automatically choose what’s newest.

Adoption also varies widely, both across organizations and within them. In my experience, platforms like X and LinkedIn tend to amplify the cutting edge; enterprise teams are often focused on integration, governance, reliability, and proving business value. Those challenges don’t disappear with a better model.

The evaluation is interesting, but I’m wondering how much these results can really tell us about production performance. The benchmark uses fictional source packs and a relatively small number of tasks, so could the results change significantly with real-world data, longer conversations, tool use, or noisier inputs? I’d also be curious to see whether the cost and latency differences remain consistent across a much larger and more diverse workload.

Great points. That’s a fair point. These results show how the models performed under the tested conditions; they don’t establish how they’ll perform in a particular production environment. Hypothetical, complex tasks let us control the evidence and verify answers, but I agree they don’t capture the full variability or domain specificity of real enterprise workloads.

Real-world data, longer conversations, tool use, and noisy inputs could change both the model rankings and the cost and latency differences. A larger, more diverse evaluation would help determine which findings hold up. Again, this is designed as a fairly simple demonstration that introduces core concepts.

The main takeaway is the evaluation method: define your acceptance criteria, test representative workloads, and compare the configurations that meet those requirements. These results are a starting point for that process, not a substitute for production-specific validation.