Let the model run what it wrote — somewhere it can do no damage
A language model that can execute code stops guessing. It checks the arithmetic, reads the real traceback, and fixes the import it got wrong. All of that requires an execution target you are willing to hand to a probabilistic system, which means a machine that is disposable, isolated, and cheap.
Execution is what closes the loop
Without execution, a model writing code is producing a plausible artefact and you are the test harness. With execution, the loop closes: generate, run, observe, repair. The observation is what carries the information — an exit code and a traceback are ground truth in a way that no amount of self-critique is.
That makes the quality of your observation channel the thing worth engineering. Most disappointing code-execution loops fail there rather than in the model: the error was truncated in the middle, stderr was thrown away in favour of stdout, the exit code was ignored so a silent failure looked like success, or the working directory reset between calls so the model's fix was applied to a file that no longer existed.
The tool definition
One shell tool, one sandbox for the run. Tell the model the state persists — otherwise it will re-create files it already wrote.
{
"name": "run_command",
"description":
"Run a shell command in a private Linux sandbox. The filesystem and any "
"installed packages persist between calls within this conversation; the "
"working directory is /home/sandbox. Python 3, Node.js 22, git, curl and "
"build-essential are installed. Use non-interactive flags: commands that "
"wait for input will time out. Returns stdout, stderr and the exit code.",
"input_schema": {
"type": "object",
"properties": {
"command": { "type": "string", "description": "Shell command to run." },
"timeout_ms": { "type": "integer", "description": "Kill the command after this long. Default 60000." }
},
"required": ["command"]
}
}import os, requests
API = "https://sandbox-as-a-service.com/v1"
H = {"Authorization": f"Bearer {os.environ['AAS_API_KEY']}"}
MAX_CHARS = 4000 # observation budget per stream
sid = requests.post(f"{API}/sandboxes", headers=H,
json={"size": "small", "timeout_minutes": 30}).json()["id"]
def run_command(command, timeout_ms=60_000):
r = requests.post(f"{API}/sandboxes/{sid}/exec", headers=H,
json={"command": command, "timeout_ms": timeout_ms,
"cwd": "/home/sandbox"}).json()
def tail(s):
# Keep the END of each stream: that is where the real error is.
return s if len(s) <= MAX_CHARS else "...[earlier output omitted]...\n" + s[-MAX_CHARS:]
parts = [f"exit_code: {r['exit_code']} ({r['duration_ms']} ms)"]
if r["stdout"]: parts.append("stdout:\n" + tail(r["stdout"]))
if r["stderr"]: parts.append("stderr:\n" + tail(r["stderr"]))
if r["truncated"]:
parts.append("NOTE: output was truncated by the sandbox; "
"re-run redirecting to a file if you need all of it.")
return "\n\n".join(parts)
try:
messages = [{"role": "user", "content": task}]
for _ in range(12):
reply = model.chat(messages, tools=[RUN_COMMAND_TOOL])
if not reply.tool_calls:
break
for call in reply.tool_calls:
messages.append(tool_result(call.id, run_command(**call.arguments)))
finally:
requests.delete(f"{API}/sandboxes/{sid}", headers=H)# What the model sees after a failing run — compact, complete, honest
# about what was cut. This is the whole product of the execution step.
exit_code: 1 (241 ms)
stderr:
Traceback (most recent call last):
File "/home/sandbox/solve.py", line 14, in <module>
total = sum(row["amount"] for row in rows)
~~~^^^^^^^^^^
KeyError: 'amount'
# The model now has the file, the line, the expression and the key. The next
# turn is usually run_command("head -3 data.json") rather than a guess —
# which is exactly the behaviour you want and cannot prompt into existence.Three things that decide whether the loop works
State persists across calls
The model writes solve.py, runs it, edits it, runs it again. If each tool call starts from a clean image, none of that works and the model compensates by regenerating the entire file every turn — burning tokens and losing fixes. One sandbox per conversation, many exec calls against it.
Errors arrive intact
Return stderr and exit code, not just stdout. Keep the tail rather than the head. Mark truncation explicitly. A model reasoning about a traceback whose last line was cut will invent a plausible cause and confidently fix the wrong thing.
The machine is worthless
The loop is only safe if you genuinely do not care what happens to the target. That means no shared filesystem, no long-lived credentials in the environment, no network path to anything internal, and a hard expiry. Then a bad command is a wasted turn instead of an incident.
Budgeting a loop
A twelve-turn coding loop typically holds one sandbox for ten to twenty minutes: on the small size, roughly $0.022 of runtime. Against the model tokens for twelve turns of generation and observation, that is noise. The practical consequence is that optimising sandbox cost is the wrong instinct — the levers that matter are the number of turns and the size of each observation.
Where sandbox cost does appear is in fan-out: a thousand parallel evaluation runs, each with its own machine, is a different arithmetic. There billing by the second helps, because a run that finishes in two minutes and four seconds bills two minutes and four seconds. Keep an eye on the 20-sandbox concurrency limit when you design a fan-out; it is there to make a leak fail loudly.
Injected instructions are the real risk
The dangerous case is not the model writing bad code by accident — it is the model reading a document, a web page or a repository that contains instructions, and dutifully executing them. No prompt prevents that reliably. What limits the damage is that the command lands on a single-tenant VM with no privileges, no metadata access, a hard expiry, and only the credentials you scoped to this one task. See secure code execution for the full model.
Questions
Why not use the code interpreter built into the model provider?
If it fits, use it — it is less work. The limits people hit are the ones you cannot change: no network access in some implementations, a fixed package set, no way to clone a private repository, no control over the machine size, and results that are hard to route into the rest of your system. A sandbox you create yourself is a normal Linux box, so the answer to "can it do X" is "install X".
Should I give the model one tool or several?
One tool that takes a shell command covers everything and keeps the schema small, which matters because every tool definition costs context on every turn. Models handle a single run_command tool well, especially if the description tells them the working directory persists between calls. Add a second tool only for something the shell genuinely cannot express, such as returning a binary artefact.
How much of stderr should I feed back?
The tail, not the head. Python tracebacks put the useful line last; compilers and test runners bury the first real error under a wall of context. A practical default is the last few thousand characters of stderr plus the exit code, and to say explicitly in the tool result when output was truncated so the model does not reason about a partial traceback as though it were complete.
What does one iteration cost?
Sandbox time is billed for the time it runs, so a short iteration costs one minute of the size you picked — $0.0015 on the small size. The expensive part of a code-execution loop is almost always the model tokens, not the machine. That changes the optimisation: keep the sandbox alive across iterations rather than recreating it, and spend your effort shrinking the output you feed back rather than shaving seconds off runtime.
Can the model be tricked into running something malicious?
Yes, and you should plan for it rather than try to prevent it in the prompt. If the model reads a web page, a repository, or a document that contains instructions, those instructions can end up as a command. The mitigation is not a better system prompt; it is that the command runs on a machine that holds nothing valuable, is destroyed shortly, and has no credentials beyond what that one task needed.
How do I make the same code produce the same result twice?
Pin everything the sandbox does not pin for you: a lockfile or explicit versions for dependencies, a fixed random seed, and an explicit base image state captured in your setup command. Sandboxes start from the same base, but pip install pandas today and in three months are different pandas. If reproducibility matters, record the setup command alongside the result.
Close the loop
One API key, one sandbox, and your model can check its own work. $${SIGNUP_BONUS} of credit on signup — about ${SIGNUP_BONUS_HOURS} sandbox-hours.