From Hypothesis to Understanding: A Practical Framework for Engineering Experiments

A working demonstration shows what’s possible; an experiment tests whether it’s worth pursuing. Before investing further, we need to know whether the proposed approach solves a worthwhile problem, improves on a competent baseline, and meets the requirements that matter.

I use the following framework most often when working with AI and agentic systems. It turns those questions into testable hypotheses and controlled experiments, using the results to decide whether to continue, change direction, or stop.

Figure 1. Eight steps from hypothesis to conclusion. Findings can lead to another experiment, a revised hypothesis, or a decision to stop.

You will often need more than one experiment. Each one helps you decide what to test next: run the experiment, analyze the results, adjust the question or design, and test again. The overall framework keeps you focused on the problem, the budget, and what you can reasonably conclude. Agree on the plan before each comparison, then keep it fixed while collecting results. Change the next experiment when the findings call for it.

Figure 2. The experimentation loop: execute the experiment, analyze the results, interpret the findings, and determine the next step. Findings can lead to a revised hypothesis, design, or protocol for another experiment, or to a conclusion that ends the investigation.

Reusable Codex Skill

Toward the end of the post, we’ll turn the framework into a reusable Codex skill for designing experiments and reviewing results in your own projects.

https://github.com/garystafford/engineering-experiments-codex-skill

The project behind the examples

We will review a series of recent agent self-improvement experiments to show how it works in practice. The agent self-improvement project explored whether a model could use feedback from a worker’s executions to write instructions that improved that worker’s performance. The initial comparisons across five small exploratory experiments followed the same method: collect baseline results, generate one replacement instruction set, and compare that frozen replacement with the original on held-out cases.

Our agent self-improvement project explored whether a model could use feedback from a worker’s executions to write instructions that improved that worker’s performance.

We began with a refund workflow and progressed to multi-turn retail customer service involving orders, payments, inventory, and policy constraints. We checked whether the worker completed the required tasks, attempted any prohibited actions, and used less money or time. The baselines met the business requirements in the completed evaluations, leaving no observed accuracy gap for the revisions to close. Some revisions reduced cost. The investigation did not establish sustained, autonomous self-improvement.

Figure 3. Experiment 1 architecture: one refund worker uses a local MCP server backed by a fresh SQLite database for each task. The Python harness collects development traces and grades, requests one replacement instruction set, and compares it with the baseline on held-out cases using an independent deterministic grader.

Experiment 1, above, tested a focused refund workflow. The results led us to examine cost more directly and, in subsequent experiments, introduce more complex operational requirements. By experiment 4, the worker coordinated orders, payments, inventory, and support across four simulated services. The framework stayed the same; the findings shaped the next question and workload.

Figure 4. Experiment 4 architecture: one worker coordinates four simulated services through local MCP connections, each with its own SQLite database. The Python harness handles API tool calls, restarts and replays tests, and independently grades business effects.

The eight-step framework

The experiments above provide the running example. The eight steps below explain how to formulate a hypothesis, design and execute an experiment, interpret the findings, and decide what to investigate next.

1. Formulate the hypothesis

State the thesis you want to investigate, then formulate a testable hypothesis: a specific change should produce a predicted outcome under stated conditions. Identify competing explanations and the results that would challenge your prediction.

Our broader thesis concerned agent self-improvement. We tested a narrower hypothesis: a model-generated revision of instructions could improve task outcomes or efficiency relative to the original instructions. An unchanged outcome or higher cost would challenge the practical benefit of that revision.

Also identify the study’s purpose. An exploratory experiment helps you determine whether an idea works well enough to warrant further investigation. A confirmatory study requires a design and sample size suited to testing a more precise claim. Exploration need not become a full trial if its findings do not justify one.

2. Design the experiment

Define the requirements before selecting the implementation: what the system must do, what it must never do, when it should ask for help, and the conditions it needs to handle. With those requirements established, define the intervention, comparison condition, controls, architecture, and evaluation environment. Selecting a baseline means choosing the starting version and measuring its performance during execution. Consider alternatives and explain what the selected design leaves untested. For example, a single-agent experiment does not show how a multi-agent design would perform.

In our experiment 5, the requirements included handling changing customer instructions, multiple orders, split payments, and exchanges. We implemented one worker connected to simulated retail services, using Python, MCP, and SQLite. That exercised a customer-service workflow while leaving multi-agent coordination and production integrations untested.

Separate what you intend to change from what must remain fixed. Otherwise, a favorable result may leave you unable to explain which change helped. In our comparisons, we changed the instructions while holding the model, tools, policies, grader, and execution limits fixed.

Compare against a baseline that already works well. Weakening it to create room for improvement can make the experiment less useful for the engineering decision you actually face.

3. Specify the protocol

Turn the experimental design into a protocol that someone else could review and implement. The design defines the comparison; the protocol specifies how to run and evaluate it. Include:

• The question, your prediction, and the decision the experiment should help you make.
• The tasks, the baseline, the proposed change, and what stays the same.
• How you will measure correctness, safety, cost, and speed, and which results will be useful.
• Which cases you will use, how often you will run them, which you will keep out of development, and how you will reset the test environment.
• The tools, estimated cost, spending cap, and limits on each run.
• How you will check the test setup, what records you will save, and what the experiment can and cannot tell you.

Explain why the run count and thresholds fit the purpose. A target for deciding whether to continue is different from proof that a result is statistically reliable. A smaller gain can still be a gain, even if it falls short of your target.

Put the limits into the implementation so they actually stop execution. Budget for development and evaluation as well as successful executions. Agree on paid work before running it, and specify how execution stops when limits are reached.

In experiment 5, we used 12 conversations to inform the revision and held out 12 new scenario versions to compare against the baseline. Each held-out case ran once per version, with the test environment reset each time, alternating which version ran first. A shared budget cap covered development and evaluation, with separate limits for requests, tool calls, and time. This was enough for an initial comparison, but not to establish how reliably the system would perform.

4. Validate the experimental setup

Before running a new experiment, validate the tools, graders, test data, and starting conditions. Unit tests can check individual tools and grading rules; integration tests can check the complete workflow. Confirm that the tasks are feasible, that known correct and incorrect outcomes receive the expected grades, and that each run starts with an isolated state.

In our implementation, the grader inspected actual refunds, exchanges, balances, and stock, as well as logs of prohibited actions. A plausible final answer alone could not pass. A faulty grader could make an effective intervention look ineffective, or vice versa.

5. Execute the experiment

Follow the agreed protocol. Keep the model configuration, test conditions, and intervention fixed during evaluation. Collect outcomes, traces, timing, usage, and costs. Record amendments rather than rewriting the original protocol afterward.

Our initial improvement comparisons followed this sequence:

1. Collected baseline execution results, traces, grades, and costs.
2. Gave the model that evidence and the baseline instructions.
3. Asked it to generate one complete replacement instruction set.
4. Froze that replacement.
5. Compared baseline and replacement on fresh held-out cases.

That sequence is how we tested instruction improvement. Another investigation might compare models, retrieval strategies, or architectures instead.

6. Analyze the results

Calculate the predefined metrics and compare them with the prediction, baseline results, and acceptance criteria. Keep the measurements separate from explanations and adoption decisions. Report worker execution costs separately from development and instruction-generation costs. Look at individual cases, averages, and totals. Separate system failures from test-setup faults, and count the cost of unsuccessful work. Judge the result using the measures you chose beforehand.

In our investigation, a revision made fewer tool calls but increased cost per successful task. That distinction mattered because the objective was to reduce cost. Another revision produced a modest cost saving but didn’t meet our target for continuing. We needed to report both the savings and that it fell short of our target.

7. Interpret the findings

Assess whether the results support the hypothesis, challenge it, or leave it unresolved. Consider alternative explanations, measurement validity, uncertainty, and the conditions to which the findings might transfer. Interpretation explains what the measurements mean; the next step states the conclusion and the action that follows.

A small test with simulated tasks can help you decide what to try next. It cannot establish production reliability. A change may perform better without the test explaining why, and testing one architecture tells you little about alternatives. In our investigation, the baselines passed the tested business requirements. We therefore observed no baseline accuracy failures for the replacement instructions to fix. That changed what we wanted to investigate next; it did not disprove self-improvement in general.

Experiment 5 used new data variants within known simulated scenario families, with one execution per case and version. That supports an observation about these cases, not a claim of production reliability or repeatable savings across other workloads. Two successful revision steps also do not establish a sustained or autonomous improvement process. If the results challenge your explanation, change it or consider an alternative. Test that new explanation on fresh cases before making a stronger claim.

8. Conclude and determine next steps

Answer the original research question. Within the tested scope, does the hypothesis remain supported, need revision, or warrant rejection? An inconclusive finding is also a conclusion. State that outcome before deciding whether to continue, redesign, investigate an alternative, run a larger study, or stop. A small exploratory study rarely justifies declaring a broad thesis proved or disproved.

Our conclusion was narrower than the original thesis: feedback-driven instruction revision sometimes reduced execution cost while preserving the tested business outcomes. We demonstrated no accuracy gain and did not establish sustained, autonomous self-improvement.

The higher cost of the first experiment prompted a more explicit cost objective in the next. Later experiments added more realistic tasks, including customer conversations with several exchanges. When the experiment-5 baseline passed its accuracy evaluation, we agreed to investigate efficiency. The second revision asked whether another improvement step could reduce cost again while retaining correctness. Each result changed what the next experiment needed to tell us.

A result that falls short of your target may still justify another experiment. Explain what you expect to learn next, why it is worth the additional time and cost, and how it follows from the findings.

Where this framework fits

The hypothesis–experiment–analysis–revision cycle is established scientific practice. OpenAI’s Evaluation best practices apply that discipline to AI systems through explicit objectives, datasets, metrics, and repeated evaluation. Its shared playbook for trustworthy evaluations further emphasizes matching the evaluation setup to the claim and checking whether the results support it.

Complementary perspectives include AI Agents That Matter, on cost, holdouts, and reproducibility, and Anthropic’s Demystifying evals for AI agents, on trials, graders, and outcomes.

Research automation has precedents too: Sakana AI’s The AI Scientist connects idea generation, experiments, and reporting; Google’s Towards an AI co-scientist develops and refines research hypotheses.

This framework applies that established cycle to engineering decisions. Before your next experiment, write down the decision it should inform. Afterward, keep your original prediction alongside the result, and explain why the next step follows.

Put the framework to work as a skill

To make the framework reusable, we packaged it as an engineering-experiments skill for Codex: written instructions supported by an experiment-brief template, a results-summary template, a decision-record template, and a worked example. The skill follows the same eight steps and offers three entry points: designing an experiment, executing an agreed protocol, or reviewing existing results.

Figure 5. The engineering-experiments skill package: SKILL.md defines the workflow, agents/openai.yaml supplies Codex metadata, assets/ contains the experiment templates, and references/ holds the worked example. Distributing the complete folder preserves the relative links between these files.

Once installed, the skill appears as Engineering Experiments in Codex’s Skills view. Its details show the workflow and supporting resources, so you can inspect the instructions before using it in a conversation.

Figure 6. Engineering Experiments installed and enabled in Codex. The detail view displays the skill’s instructions, design, execute, and review entry points, and links to templates and worked examples.

In the desktop app, type @ and select Engineering Experiments. In the Codex CLI or IDE extension, use $engineering-experiments or select it through /skills. The examples below use $, but both forms select the same skill.

Here, in the Codex desktop app, I ask for a protocol, run count, and budget for a caching experiment, explicitly limiting the task to design. Codex returns a proposal whose assumptions and acceptance criteria I can review and refine before authorizing execution.

Figure 7. Using Codex’s engineering-experiments skill to design a caching experiment. The response proposes a hypothesis, comparison conditions, and evaluation criteria, and states that it has not executed any runs or paid requests.

The same skill is available in the Codex CLI. Here, I ask whether adding a review pass can reduce factual errors enough to justify the extra latency and cost. The request remains at the design stage, allowing me to assess the proposed workload and evaluation effort before committing to execution.

Figure 8. Invoking the engineering-experiments skill in the Codex CLI to plan a review-pass experiment. Codex reads the skill and experiment-brief template, then proposes a run count, API budget, and human evaluation effort. It identifies these as proposed design choices and confirms that nothing has been executed.

Using the skill, you can enter at the stage you need rather than restart the whole process. For example, a design request could be:

Use $engineering-experiments to turn this hypothesis into a bounded experiment brief. Identify the baseline, architecture alternatives, measures, run count, budget, and limits on the conclusions. Flag unresolved choices before execution.

After collecting results, a review request could be:

Use $engineering-experiments to review these results against the original brief. Separate observations from explanations, identify what remains uncertain, and recommend whether to stop, revise, or run a fresh experiment.

The skill’s purpose is practical: help you apply the approach to your next experiment and produce a plan or review you can use. Goals, spending authorization, and the claims made from the results still require human judgment.

This blog represents my viewpoints, not those of my past or current employers. All product names, images, logos, and brands are the property of their respective owners.