Undertow: A Synthetic Benchmark Experiment for Programmatic Tool Calling

A small experiment in building controllable hidden tool-use worlds, and using traces to understand where agents fail.

TL;DR

I built Undertow because I wanted a simple environment where I could ask the question:

When an agent has to control a hidden, stateful world through primitive tools, does having a programmable workspace help it solve tasks more reliably than a standard tool-based environment?

This is meant to be a controlled task. The point is to know exactly what the agent is given, exactly what is hidden, exactly what actions it can take, and exactly why it succeeds or fails.

Undertow currently has two task families:

• hidden_dag: the agent gets initial values, generated function schemas, and a target variable. It has to call functions in a valid dependency order and submit the target value.
• repair_chain: the agent gets machine ids, tool ids, bay ids, and primitive operation schemas. It has to inspect tools, scan machines, move to the right bay, apply the right repair tool at each stage, track stale observation versions, test repaired machines, and submit the repaired ids.

This benchmark makes hidden-state failures visible. It tells me whether an agent failed because it forgot a value, called a function too early, used the wrong repair tool, reused a stale observation, repeated calls, hit a constraint, timed out, or produced metrics that disagree with each other.

Flow

1. Motivation
2. Why synthetic at all
3. What the agent is given
4. Task 1: hidden_dag
5. Task 2: repair_chain
6. What PTC changes
7. What the results say so far
8. What went wrong in the traces
9. What PTC actually did
10. Closing

Motivation

The motivating intuition is that ordinary tool calling and programmatic tool calling are not very different on short tasks. If a task is "call one API and report the result," a Python workspace should not matter much.

The workspace should matter when the agent has to build and maintain state over many tool calls. For example:

• a table of values it has already computed,
• a queue of functions whose inputs are now available,
• a map from machine to location,
• a map from (part, repair_stage) to tool,
• the current observation version for each machine,
• and a final value or repaired-object list that has to be consistent with the world.

This led to the question:

Can I generate tasks where the correct strategy is to build a small local state machine around primitive tools?

If yes, then PTC has a natural thing to be good at: writing that state machine down in code. If no, then the benchmark is not actually testing what I want it to test.

So the current version of Undertow is best understood as an environment-design experiment.

Why Synthetic At All

The reason to make Undertow synthetic is controllability. In a real tool-use task, many things vary at once: API surface, domain knowledge, prompt wording, hidden data. In Undertow, I can generate a world from a seed and turn the knobs.

That lets me ask questions like:

• Does success fall when the dependency chain gets deeper?
• Do connected distractor functions cause more wasted calls than disconnected distractors?
• Does stale state break agents even when the correct next action is obvious?
• Does the workspace help because the agent writes a real ledger, or does it just make different mistakes?
• Once the agent discovers the general strategy, do bigger generated instances still matter?

That last question is important. If the knobs stop mattering once the agent figures out the strategy, that is a useful negative result. It means the benchmark is mostly testing strategy discovery, not scalable execution. If the knobs continue to matter, then the benchmark is also testing whether the agent can carry out the strategy reliably as bookkeeping pressure increases.

This is the same reason I found FuncBenchGen useful as a reference point. FuncBenchGen is also synthetic, but it is clear about the generated object: a hidden function-dependency DAG. The model sees functions and input variables, but not the graph. The task is to compute a target value. The knobs change graph size, depth, and distractors.

Undertow borrows that style of benchmark design and asks whether it can be extended from hidden dataflow to hidden mutable state.

What The Agent Is Given

The agent receives a public task spec and primitive tools.

In standard mode, the system prompt says the agent is solving an Undertow task with native tools. The user prompt says:

Discover the target answer using the primitive tools.
Finish with submit(...).

Public task spec:
{...}

In PTC mode, the system prompt says the agent is in a programmatic tool-calling environment. It can use Python files under /workspace, and it can call the same primitive tools through:

from ptc import task, tools, finish

The PTC prompt also gives the public task spec. It does not expose hidden state or reward.

So yes, the model is told the task family through the public spec. For example, it can see "family": "hidden_dag" or "family": "repair_chain". But it is not told the hidden solution:

• In hidden_dag, it sees function inputs and outputs, but not the true value table or the already-computed solution order.
• In repair_chain, it sees machine ids, tool ids, and bay ids, but not each machine's required repair sequence.

The benchmark is not testing whether the model can guess that there is structure. It is testing whether the model can use the public structure and tool observations to maintain the right state over time.

Task 1: hidden_dag

hidden_dag is the simpler family. It is basically a generated dataflow puzzle.

The agent sees something like:

{
  "family": "hidden_dag",
  "initial_values": {
    "v0": 343,
    "source_3": 706
  },
  "target": "v2",
  "functions": [
    {
      "name": "core_0",
      "inputs": ["v0"],
      "output": "v1",
      "description": "Transforms v0 into v1."
    },
    {
      "name": "core_1",
      "inputs": ["v1", "source_3"],
      "output": "v2",
      "description": "Combines v1, source_3 into v2."
    }
  ]
}

It can call core_0 only if it passes the correct current value for v0. A successful call returns a new variable:

{
  "ok": true,
  "variable": "v1",
  "value": 596
}

Now the agent can call core_1, because it knows both v1 and source_3. If it submits the correct value for v2, the hidden checker marks the task solved.

The intended strategy is:

known = dict(initial_values)
pending = list(functions)

while target not in known:
    for fn in list(pending):
        if all(x in known for x in fn["inputs"]):
            result = call(fn["name"], **{x: known[x] for x in fn["inputs"]})
            known[result["variable"]] = result["value"]
            pending.remove(fn)

submit(value=known[target])

This is why hidden_dag is a good first family. It is not trying to be realistic. It is trying to isolate one boring but important behavior: can the agent maintain a value ledger and execute a dependency graph without losing track?

The knobs mean:

If an agent has the right topological-execution strategy, increasing n_core and depth should still matter because there are more exact values to carry forward. If it does not matter, then hidden_dag is probably too easy for that model/interface pair.

Task 2: repair_chain

repair_chain is the family that actually motivated Undertow. Here the agent is controlling a hidden mutable world, not just computing a value.

The public spec looks more like:

{
  "family": "repair_chain",
  "objects": ["machine_0", "machine_1"],
  "locations": ["bay_0", "bay_1"],
  "tools_available": ["tool_0", "tool_1", "distractor_tool_0"],
  "target": "repair all critical machines"
}

The agent can use these tools:

A typical observation might be:

{
  "ok": true,
  "object_id": "machine_0",
  "status": "broken",
  "part": "part_0",
  "location": "bay_1",
  "repair_stage": 0,
  "observation_version": 0
}

The key detail is observation_version. When the agent repairs a machine, the version can change. If the agent later uses the old version, the environment returns stale_state_reference. This is a tiny synthetic version of a real state-control problem: what you observed earlier may no longer be valid after you act.

The intended strategy is:

tool_map = inspect_all_tools()      # (part, stage) -> tool_id
machines = scan_all_machines()      # machine_id -> current state

for machine_id, machine in machines.items():
    move_to(machine["location"])

    while machine["status"] != "repaired":
        tool_id = tool_map[(machine["part"], machine["repair_stage"])]
        result = apply_tool(
            object_id=machine_id,
            tool_id=tool_id,
            observation_version=machine["observation_version"],
        )
        machine.update(result)

    test_object(machine_id, machine["observation_version"])

submit({"repaired": sorted(machines)})

The knobs mean:

This is where I expect PTC to matter more. In standard mode, the agent has to carry the ledger through the conversation. In PTC mode, the agent can write the ledger into code and files.

But this is still a hypothesis. The current results do not yet prove it.

What PTC Changes

The two modes get the same public task data and the same primitive environment tools.

The difference is the control surface.

In standard mode, the agent directly calls tools one at a time:

scan_object(machine_0)
inspect_tool(tool_0)
move_to(bay_1)
apply_tool(machine_0, tool_0, observation_version=0)
...
submit(value=None, repaired=["machine_0"])

In PTC mode, the agent can write code around those calls:

from ptc import task, tools, finish

spec = task.spec()
machines = {}
tool_map = {}

for tool_id in spec["tools_available"]:
    obs = tools.inspect_tool(tool_id=tool_id)
    # store useful compatibility information

finish({"repaired": repaired_ids})

PTC does not make the hidden state visible. It only makes it easier for the agent to build durable local state: caches, ledgers, queues, assertions, logs, and final answer constructors.

So the actual claim is:

PTC should help if the benchmark requires reliable local state management around external tool calls.

Undertow is my attempt to build tasks where that claim is testable.

What The Results Say So Far

The current results should be treated as preliminary diagnostics, not a final benchmark result. The result set is small, and the bigger configs are stress probes rather than balanced multi-seed sweeps.

On easy, hidden_dag is basically solved by everyone. All three models solve it in both modes. That makes it useful as a calibration check, but not as the main story. repair_chain is already less uniform: standard solves 2 of 3 runs, while PTC solves 1 of 3. DeepSeek V4 Flash fails both ways, but the failures are different. In standard mode it burns the whole 165-call budget, with 145 failed calls, including many unknown-state and constraint failures. In PTC mode it fails earlier, at 70 calls and 50 failed calls, but still does not build enough useful repair progress. Qwen3.6 35B A3B standard solves the easy repair task despite 116 failed calls; Qwen3.6 35B A3B PTC reaches only 0.24 progress and spends most of its failed calls repeating work.

On medium, the benchmark starts to show the distinction I care about. hidden_dag standard is almost boring: 6 of 6 solved, 1.0 progress, and no failed calls. It behaves like a topological executor. PTC can also get through the family, but the traces are noisier. Qwen3.6 35B A3B PTC solves both seeds while producing schema errors, repeated calls, and incorrect-value calls. DeepSeek V4 Pro PTC has rows where progress=1.0 and reward=1.0 but solved=false. I would not narrate that as a model misunderstanding the task. I would treat it as a benchmark instrumentation issue to inspect, because the metrics are not telling the same story.

The most useful comparison is medium repair_chain. DeepSeek V4 Flash solves both seeds in both modes, but standard seed 1 is much tighter: 213 calls, 16 failed calls, reward 1.0. The matching PTC run solves too, but takes 366 calls and 155 failed calls. DeepSeek V4 Pro shows the other side: on seed 0, standard stalls at 0.081 progress, while PTC reaches 1.0 progress and solves, even though it takes 424 calls and 212 failed calls. Qwen3.6 35B A3B is split again. PTC solves seed 0 where standard only reaches 0.6875 progress, but PTC collapses on seed 1 with 0.03125 progress. I would read this as "PTC changes the failure surface," not "PTC is better."

On hard, the run becomes more like a stress test than a balanced comparison. The completed hidden_dag rows still mostly work, which suggests that dataflow bookkeeping alone is not the hard part for these models. repair_chain is different. The standard hard repair-chain runs complete, but barely progress: DeepSeek V4 Flash reaches 0.027 progress, DeepSeek V4 Pro reaches 0.031, and Qwen3.6 35B A3B reaches 0.0059. Those traces are mostly constraint violations and repeated calls. The PTC repair-chain attempts for hard do not have aggregate rows because the Python runner timed out after 600 seconds.

I would not read v_hard as a result yet. The aggregate CSVs are empty and there are no raw rows or logs. It is a configured stress target, not evidence.

I would read this as a map of trajectories, not as a winner chart.

What Went Wrong In The Traces

Two failed runs can mean very different things:

• The agent never discovers the right state.
• The agent makes partial progress and then stalls.
• The agent reaches full progress, but reward, progress, and solved disagree.
• The agent repeats calls until most of the budget becomes noise.
• The agent uses stale or unknown observation versions after state changes.
• The agent violates environment constraints, like using the right tool in the wrong place or at the wrong stage.
• The agent runs a local program long enough to hit the sandbox timeout.

Undertow records these differences.

The useful thing here is that repeated calls, constraints, stale state, and timeout-like failures become separate facts.

For hidden_dag, the standard trajectory is usually simple: compute the functions whose inputs are available, update the value ledger, and submit the target. Across the completed rows, standard mode does this almost exactly. The PTC traces are more interesting because the workspace is sometimes extra surface area. A script can put a wrong value into the local ledger, retry a function with bad inputs, or call a generated function through the wrong schema. That is why some PTC hidden_dag runs have full progress but noisy failure counters.

hidden_dag mostly checks whether the agent can act like a topological executor.

For repair_chain, the useful question is more mechanical: did it scan all machines, inspect enough tools, move to the right bay, apply the right tool at the right observation version, update after each state change, and avoid doing the same thing again?

This is where the traces become useful. In the medium runs, repeated calls dominate the failure counts. Across all medium episodes, there are 1,378 repeated calls. PTC repair-chain runs are especially repeat-heavy: DeepSeek V4 Flash PTC repair has 330 repeated calls, DeepSeek V4 Pro PTC repair has 358, and Qwen3.6 35B A3B PTC repair has 260 repeated calls plus 16 stale-state references. Standard repair-chain runs also repeat, but their failures often look more like a slow conversational stall: partial progress, many turns, and eventually no useful next move.

repair_chain is where success, progress, and call discipline start to separate.

Call efficiency and progress make the same point. A benchmark score should not only say whether an agent won. It should say how much of the interaction actually changed the world in the right direction.

Useful-call rate asks whether tool calls are becoming progress or disappearing into retries and mistakes.

Progress and success can diverge. The current disagreement between progress, reward, and solved status is something to fix, not something to hide.

Call counts show how much budget agents spend before solving, failing, or giving up.

Budget is also part of the environment design. If the budget only allows the perfect oracle path, then the task mostly tests whether the model immediately guesses the intended algorithm. If the budget is too loose, repeated calls stop mattering. Undertow treats budget as another knob.

Budget calibration controls how much exploration and recovery the benchmark permits.

What PTC Actually Did

The best PTC traces look like the strategy I wanted the benchmark to reward. The agent scans objects, inspects tools, builds a local map from (part, stage) to tool, loops over machines, and updates its local state after each repair. That is what happens in some medium repair_chain runs where PTC recovers cases that standard leaves partial. DeepSeek V4 Pro seed 0 is the clearest example: standard stops at 0.081 progress, while PTC reaches full progress and solves.

But PTC also makes a bad local policy fast. If the script retries blindly, carries a polluted state map, or does not stop after repeated errors, it can burn calls much faster than a conversational agent. That is what the medium repair-chain failure counts show. The workspace helps when it becomes a ledger and planner. It hurts when it becomes an unchecked retry loop.

The hard repair-chain timeouts are another version of the same point. Those PTC runs did not produce aggregate rows because the Python command timed out after 600 seconds in the sandbox. That is not a normal solved/unsolved benchmark score, but it is still useful information. At larger sizes, the generated local program has to manage its own runtime.

On hidden_dag, PTC is less obviously useful. Standard mode already behaves like the intended topological executor. In that family, the workspace mostly creates new ways to make small mistakes: schema calls, repeated calls, and value-ledger pollution If the task is already easy to do directly, adding a programmable layer does not automatically help.

Closing

PTC should help when a tool-use task can be converted into a local state machine. Undertow is a controlled, synthetic attempt to generate hidden tool-use worlds where that claim can be tested—and where state tracking, recovery, repeated calls, stale observations, constraint failures, timeouts, and metric disagreement can be measured separately.

Acknowledgements

Grateful to @a1zhang for detailed feedback on early versions of the write-up and for helping me scope the research problems.

And of course, the @PrimeIntellect team for the generous compute credits.