Goal: hand the harness a failing test run and get back a fix you approved, plus two or three sentences on what broke.
Think of the harness so far as a new hire with a badge and a supervisor. It can look around, and it asks before it touches anything. This phase hands it a first ticket: the build is red, find out why.
You don't need a real CI log for this. A CI log is the printed output of a test command that failed, so you make your own: a tiny shop module with one planted bug and a test suite that catches it. It lives inside the repo, so anyone who clones it gets the same failure you do.
Groq's free tier is a second reason to keep it small. Both gpt-oss models allow 8,000 tokens per minute (rate limits), and the harness resends the whole conversation on every step. A real project's log plus a few of its source files would hit that ceiling before the model finished reading. A 15-line module and a 900-character log stay well under it.
You'll add one new tool, one new guardrail check, and a script called doctor.py that runs the tests, hands the failure to the harness, and runs the tests again to check the fix.
1. Plant a bug in a tiny shop module
Create a folder called examples/shop inside the project folder, then create examples/shop/cart.py:
BULK_THRESHOLD = 3
BULK_DISCOUNT = 0.10
def line_total(price: float, quantity: int) -> float:
"""Price for one line of the cart. Buying three or more of an item takes 10% off that line."""
total = price * quantity
if quantity > BULK_THRESHOLD:
total = total * (1 - BULK_DISCOUNT)
return round(total, 2)
def cart_total(lines: list) -> float:
"""Sum of every line. Each line is a (price, quantity) pair."""
return round(sum(line_total(price, quantity) for price, quantity in lines), 2)
The docstring promises a discount at three or more items. The code gives it at more than three, so three items pay full price. It's the kind of off-by-one a reviewer misses, and the traceback alone won't point at the line. The model has to open cart.py and compare the code against the docstring.
Create examples/shop/test_cart.py:
import unittest
from cart import cart_total, line_total
class LineTotalTest(unittest.TestCase):
def test_one_item_pays_full_price(self):
self.assertEqual(line_total(10.0, 1), 10.0)
def test_three_items_get_the_bulk_discount(self):
self.assertEqual(line_total(10.0, 3), 27.0)
def test_five_items_get_the_bulk_discount(self):
self.assertEqual(line_total(10.0, 5), 45.0)
class CartTotalTest(unittest.TestCase):
def test_mixed_cart(self):
self.assertEqual(cart_total([(10.0, 3), (2.5, 2)]), 32.0)
if __name__ == "__main__":
unittest.main()
The tests use unittest, which ships with Python, so there's nothing new to install. Add examples/shop/README.md so a visitor to your repo doesn't open a pull request to fix the bug:
# shop fixture
cart.py has a bug on purpose. doctor.py uses it to demo the build doctor, so leave it broken.
From the project folder, with the virtual environment active, run the tests:
python -m unittest discover -s examples/shop
You should see F..F and two failures: AssertionError: 30.0 != 27.0 for three items, and 35.0 != 32.0 for the mixed cart. That printout is your CI log.
Commit the broken module now, before writing any harness code:
git add examples
git commit -m "add shop fixture with a planted bug"
Every successful doctor run fixes cart.py on disk. With the broken version committed, git restore examples/shop/cart.py puts the bug back, so you can rerun the demo as often as you like.
2. Give the model a way to edit files
write_file only creates new files, on purpose, so it can't fix cart.py. Add a tool that changes an existing file by swapping one exact block of text for another. The model quotes the lines it wants gone and supplies the lines to put in their place. Most coding agents edit files this way, because quoting the old text proves the model read the file instead of guessing at it.
In tools.py, add this function above run_shell:
def replace_in_file(path: str, old: str, new: str) -> str:
target = ROOT / path
text = target.read_text(encoding="utf-8")
target.write_text(text.replace(old, new, 1), encoding="utf-8")
return f"replaced 1 match in {path}"
Add it to REGISTRY:
"replace_in_file": replace_in_file,
Then add its schema to TOOLS, above the run_shell entry:
{
"type": "function",
"function": {
"name": "replace_in_file",
"description": "Edit an existing file by replacing one exact block of text. old must appear in the file exactly once.",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string", "description": "File path, relative to the project folder."},
"old": {"type": "string", "description": "The exact text to replace, copied from the file."},
"new": {"type": "string", "description": "The text to put in its place."},
},
"required": ["path", "old", "new"],
},
},
},
The function itself trusts its arguments. Checking them is the guardrail's job, the same split phase 2 set up for the other tools.
3. Guard the edit
This is the most dangerous tool the harness has had so far, because it changes code you already wrote. It gets three rules, all in guardrails.py.
Add it to NEEDS_APPROVAL, and add the test command to the allowlist so the model can rerun the tests itself:
NEEDS_APPROVAL = {"run_shell", "write_file", "replace_in_file"}
"git branch",
"python -m unittest discover -s examples/shop",
}
In check, send replace_in_file to its own function before the path branch:
if name == "replace_in_file":
return check_edit(args)
if name in ("read_file", "list_dir", "write_file"):
return check_path(args.get("path", "."))
Then add check_edit above check_command:
def check_edit(args: dict) -> str | None:
path = args.get("path", "")
problem = check_path(path)
if problem:
return problem
if Path(path).name.startswith("test_"):
return "test files are read-only. Fix the source code, not the test"
text = (ROOT / path).read_text(encoding="utf-8")
count = text.count(args.get("old", ""))
if count == 0:
return "old text not found. Read the file again and copy the lines exactly"
if count > 1:
return f"old text appears {count} times. Include more surrounding lines so it matches once"
return None
The path rule is phase 2's, reused. The test rule stops the cheapest fix there is: change 27.0 to 30.0 in the test and call it green. You could ask for that in the prompt, and doctor.py does, but a model can ignore a prompt. It can't get past check_edit. When a model tries it anyway, you'll see an amber blocked row in the viewer.
The match rule runs before the approval prompt. If the model misquotes a line, the guardrail sends it back to reread the file, and you never approve an edit that was going to fail.
4. Improve the approval prompt
The approval prompt prints every argument as-is. For an edit, that means two blobs of text you'd have to compare by eye. Print a diff instead, the same red-and-green view you read in a pull request. In harness.py, add import difflib at the top, then add show_diff above execute:
def show_diff(args: dict) -> str:
lines = difflib.unified_diff(
args.get("old", "").splitlines(),
args.get("new", "").splitlines(),
fromfile=args.get("path", ""),
tofile=args.get("path", ""),
lineterm="",
)
return "\n".join(lines)
Then use it in approve:
def approve(name: str, args: dict) -> bool:
print(f"\nThe model wants to use {name}:")
if name == "replace_in_file":
print(show_diff(args))
else:
for key, value in args.items():
print(f" {key}: {value}")
return input("Allow? [y/N] ").strip().lower() == "y"
A diff makes a no easy to give. The model then has to take no for an answer. Today it reads denied: the user said no as a temporary setback and asks again, sometimes with a line of whitespace changed, until MAX_STEPS runs out. Fix it on both sides: a denial message that says to stop, and a memory of what you already refused so the exact same call never reaches you twice.
Add the message above provider = get_provider():
DENIED_MESSAGE = "denied: the user said no. Do not retry this call or a variation of it. Carry on without it if you can, or stop and explain what you wanted to do and why."
Then replace execute:
def execute(name: str, args: dict, denied: set) -> tuple[str, str]:
problem = check(name, args)
if problem:
return "blocked", f"blocked: {problem}"
key = name + json.dumps(args, sort_keys=True)
if key in denied:
return "blocked", "blocked: the user already said no to this exact call. Do not ask again."
if name in NEEDS_APPROVAL and not approve(name, args):
denied.add(key)
return "denied", DENIED_MESSAGE
try:
return "ok", REGISTRY[name](**args)
except Exception as error:
return "error", f"error: {error}"
denied lives for one run. In run, add denied = set() on the line after total_tokens = 0, and pass it along in the tool loop: outcome, result_text = execute(name, args, denied). sort_keys=True makes the key the same whatever order the model lists the arguments in. A repeat comes back as blocked, amber in the viewer, without a prompt.
The message leaves room to carry on on purpose. If you say no to the model rerunning the tests, you still want it to go and fix the bug. If you say no to the fix, there's nothing left for it to do, so it stops and explains.
The message does most of the work. The memory is the lock behind it, the same split as the test-file rule in step 3.
run needs two new arguments. context carries the test output, and run_id lets doctor.py log to the same run once the harness finishes. Replace the top of run, down to the messages list:
def run(task: str, context: str = "", run_id: str | None = None) -> str:
run_id = run_id or time.strftime("%Y%m%d-%H%M%S")
log({"run_id": run_id, "event": "task", "task": task})
prompt = task
if context:
log({"run_id": run_id, "event": "context", "text": context})
prompt = f"{task}\n\n{context}"
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
]
The model sees the task and the log together. The log file keeps them apart, so the viewer's sidebar shows a one-line task instead of a 900-character wall of test output. python harness.py "..." still works as before, because both new arguments are optional.
5. Catch rejected tool calls
The gpt-oss models sometimes call a tool you never gave them. OpenAI trained them with a built-in browser tool, and a name like open_file slips out, most often a few steps into a task. Groq checks every tool call against the request's tool list and rejects that reply with a 400 error and the code tool_use_failed:
openai.BadRequestError: Error code: 400 - {'error': {'message': "Tool call validation failed: ... attempted to call tool 'open_file' which was not in request.tools", 'code': 'tool_use_failed', 'failed_generation': '{"name": "open_file", ...}'}}
That error arrives as an exception from provider.complete, before any tool call exists for the loop to handle. Phase 1's rule of sending failures back to the model as text never gets a chance, so the whole run crashes. Phase 1's troubleshooting suggested a retry, but the identical request often gets the identical mistake. Telling the model what went wrong works better.
In backend.py, import the error class and add an exception of your own above Completion:
from openai import BadRequestError, OpenAI
class ToolCallRejected(Exception):
"""The provider refused a malformed tool call before the harness saw it."""
Then wrap the request in GroqProvider.complete:
start = time.perf_counter()
try:
response = self.client.chat.completions.create(**kwargs)
except BadRequestError as error:
if error.code != "tool_use_failed":
raise
body = error.body if isinstance(error.body, dict) else {}
raise ToolCallRejected(body.get("failed_generation", str(error))) from error
latency_ms = round((time.perf_counter() - start) * 1000)
tool_use_failed is Groq's name for the problem, so the translation belongs in backend.py. harness.py only ever sees ToolCallRejected. A second provider would raise the same exception from its own error format, and the loop wouldn't change. That's phase 3's provider abstraction doing its job.
In harness.py, import the new exception:
from backend import ToolCallRejected, get_provider
Add one sentence to SYSTEM_PROMPT:
SYSTEM_PROMPT = "You are a coding assistant. When a task needs a tool, call the tool right away. Only call the tools provided in this request. Do not announce what you are about to do, and never ask the user for permission in your reply. The harness asks for approval by itself."
Then replace the result = provider.complete(messages, tools=TOOLS) line at the top of the loop:
try:
result = provider.complete(messages, tools=TOOLS)
except ToolCallRejected as error:
print(f"[{step + 1}] rejected tool call: {error}")
log({"run_id": run_id, "event": "rejected", "step": step + 1, "generation": str(error)})
messages.append(
{
"role": "user",
"content": f"Your last tool call was rejected because it named a tool that doesn't exist or had malformed arguments: {error}. Use only these tools: {', '.join(REGISTRY)}.",
}
)
continue
The note goes in as a user message because the rejected reply never reached you, so there's no tool call id to answer. continue still uses up a step, so MAX_STEPS caps a model that keeps inventing tools. The system prompt sentence makes the mistake less frequent. The handler covers the times it happens anyway.
If python doctor.py already crashed on you with this error, restore the bug with git restore examples/shop/cart.py after making these changes and run it again. You should now see a line like [2] rejected tool call: {"name": "open_file", ...} followed by a read_file call on the next step.
6. Write doctor.py
Create doctor.py in the project folder:
import subprocess
import time
from guardrails import ROOT, log
from harness import run
TEST_COMMAND = "python -m unittest discover -s examples/shop"
MAX_LOG_CHARS = 3000
TASK = (
"The test suite for examples/shop is failing. Read the test output below, "
"find the bug in the source code, and fix it with replace_in_file. "
"Then explain the cause and the fix in two or three sentences."
)
def run_tests() -> tuple[bool, str]:
result = subprocess.run(
TEST_COMMAND, shell=True, capture_output=True, text=True, errors="replace", cwd=ROOT
)
return result.returncode == 0, (result.stdout + result.stderr)[-MAX_LOG_CHARS:]
def main() -> None:
passed, output = run_tests()
if passed:
print("The tests already pass. Nothing to fix.")
return
run_id = time.strftime("%Y%m%d-%H%M%S")
context = f"Command: {TEST_COMMAND}\n\nOutput:\n{output}"
print(run(TASK, context=context, run_id=run_id))
passed, output = run_tests()
log({"run_id": run_id, "event": "verify", "passed": passed, "output": output})
print("\nTests pass now." if passed else f"\nTests still fail:\n{output}")
if __name__ == "__main__":
main()
doctor.py plays the part of the CI server. It runs the tests without the model's help, both before and after the harness gets involved. The first run produces the log. The second checks the model's claim: a model can say it fixed the bug when it didn't, and a green test run settles the question.
run_tests keeps the last 3,000 characters of output. Test runners print the summary and the final failures at the bottom, and that's where the model needs to look. unittest writes its report to stderr, which is why the function joins stdout and stderr.
7. Show the new events in the viewer
Phase 4's Timeline already shows unknown events as raw JSON, so context, rejected and verify would appear without any changes. Give them proper rows anyway. In ui/src/Timeline.jsx, add these three blocks inside Event, above the final return:
if (event.event === 'context') {
return (
<Row time={time} kind="task" title="Test output given to the model">
<details>
<summary>Show log</summary>
<pre>{event.text}</pre>
</details>
</Row>
)
}
if (event.event === 'rejected') {
return (
<Row time={time} kind="blocked" title={`Step ${event.step}: tool call rejected by the provider`}>
<code>{event.generation}</code>
</Row>
)
}
if (event.event === 'verify') {
return (
<Row time={time} kind={event.passed ? 'ok' : 'error'} title={event.passed ? 'Tests pass' : 'Tests still fail'}>
<details>
<summary>Test output</summary>
<pre>{event.output}</pre>
</details>
</Row>
)
}
All three reuse existing CSS classes: blue for the log, amber for a rejected call, green or red for the verdict. runs.js needs no change. It takes a run's status from its answer line, and verify arrives after that.
8. Run it
Start the viewer in the second terminal (npm run dev inside ui) and open http://localhost:5173. In the first terminal, from the project folder:
python doctor.py
The model's path varies from run to run, but a good one looks like this:
[1] read_file {"path": "examples/shop/cart.py"} -> ok
The model wants to use replace_in_file:
--- examples/shop/cart.py
+++ examples/shop/cart.py
@@ -1 +1 @@
- if quantity > BULK_THRESHOLD:
+ if quantity >= BULK_THRESHOLD:
Allow? [y/N] y
[2] replace_in_file {"path": "examples/shop/cart.py", ...} -> ok
openai/gpt-oss-120b: 3912 tokens, 2140ms total
The bulk discount used > instead of >=, so three items paid full price...
Tests pass now.
Read the diff before you type y. Phase 2 built that prompt so the model can propose a change and only you can apply it. The model may also read test_cart.py first, try to edit it and get blocked, or ask to rerun the tests. All three are fine, and each one shows up as its own row in the browser.
In the viewer, the run reads top to bottom: the task, the test log under Show log, the files the model read, the edit with its outcome, the answer, and a green Tests pass row.
Now put the bug back and try the other path:
git restore examples/shop/cart.py
python doctor.py
Answer n this time. The model reads the denial, stops calling tools, and explains the fix it wanted to make. doctor.py then prints Tests still fail. If the model asks for the identical edit again anyway, the harness blocks it without prompting you. The viewer ends on an amber denied row, the answer, and a red Tests still fail row. Run git restore examples/shop/cart.py again when you're done, so the next run starts from red.
Done when: python doctor.py goes from red to green with one y from you, and the viewer tells the whole story without runs.jsonl open. Then show it to someone who hasn't seen the project. If they can say what problem it solves after a five-minute demo, without you narrating the code, the phase is done.
9. Troubleshooting
If both runs in step 8 behaved as described, skip to step 10. Otherwise, match the symptom:
- The run crashes with
openai.BadRequestError: Error code: 400andtool_use_failed, often naming a tool likeopen_file. The model called a tool the harness never offered, and nothing caught Groq's rejection. Add step 5's changes tobackend.pyandharness.py, then restorecart.pyand run again. The tests already pass. Nothing to fix.An earlier run fixedcart.pyand you haven't restored it. Rungit restore examples/shop/cart.py.- After you answer
n, the model asks for the same edit again and again until the step limit.executeis missing thedeniedset from step 4, orharness.pystill returns the olddenied: the user said notext. Check both. A small change in whitespace makes a new call, so you may still see one or two slightly different requests before the model stops. git restoresays pathspec did not match. The fixture was never committed. Go back to the end of step 1 and commit it, with the bug still in place.Error code: 429withrate_limit_exceededand tokens per minute in the message. The conversation grew past Groq's 8,000 tokens per minute. Theopenaiclient retries twice on its own, so a single 429 often passes. If the run still fails, wait a minute and run it again, and keepMAX_LOG_CHARSat 3000 or lower.- The model keeps getting blocked with
old text not found. It's retyping the line from memory instead of copying it, often with the wrong indentation. The guardrail's message tells it to reread the file, and it usually recovers on the next step. If it burns through all 10 steps, run it again.openai/gpt-oss-120bdoes better at this than the 20b model. - The model tries to edit
test_cart.pyand gets blocked. The guardrail is working. Leave the rule in place, even if the model burns a step on it. run_shellcomes back blocked when the model tries to run the tests. It typed a variation of the command, likepython3orpytest, and the allowlist only takes exact matches.doctor.pychecks the result itself, so the fix still counts. Add the variant toALLOWED_COMMANDSif it keeps happening.- The test output says
No module named cart. The test command ran from the wrong folder.TEST_COMMANDhas to run from the project folder, which is whatcwd=ROOTinrun_testsdoes. Check that line is there. TypeError: run() got an unexpected keyword argument 'context'.harness.pystill has phase 4'sdef run(task: str). Replace it with the version from step 4.- The viewer shows
contextorverifyas raw JSON. The new blocks inTimeline.jsxsit below the finalreturn, where they never run. Move them above it.
Stuck?
Ask an agent something like: "My build doctor applied an edit but the tests still fail. Here are the runs.jsonl lines for that run and cart.py after the edit. Did the model fix the wrong line, or did my replace_in_file write something unexpected?" Paste the lines for that one run_id, not the whole log.
10. Commit and push
Restore the bug first, because the repo should ship broken for the next person:
git restore examples/shop/cart.py
git status
You should see doctor.py as new, and backend.py, harness.py, tools.py, guardrails.py and ui/src/Timeline.jsx as changed. examples/shop/cart.py should not appear. If it does, the restore didn't take.
git add .
git commit -m "phase 5: build doctor with guarded file edits"
git tag v0.6-build-doctor
git push origin main --tags
This tag follows phase 4's v0.5-observability and completes the v0.1 through v0.6 set from the publishing checklist. Run git diff v0.5-observability v0.6-build-doctor --stat to see which files the phase touched.
Where to take it next
The fixture proves the mechanism. These three take it further, in rough order of effort:
- A real CI log. Add a GitHub Actions workflow that runs the same
unittestcommand on every push. It fails, since the bug is committed, andgh run view <run-id> --log-failedprints the failing step's log. Givedoctor.pya--logoption that reads a saved log instead of running the tests, and the harness is reading output from an actual CI server. - A forked project. python-slugify is MIT-licensed, about one module of real code, and runs its tests with
python test.py. Fork it, break something on a branch, and point the doctor at it. You'll need to makeROOTinguardrails.pypoint at the fork instead of the harness folder, and expect 429s: its test file alone is longer thanMAX_CHARS. - The onboarding assistant from the original plan. Keep only
read_file,list_dirandwrite_file, and ask the harness to write anONBOARDING.mdfor a repo it hasn't seen. It's read-only apart from one new file, so it shows the harness from a safer angle.
Pick one if you want more. The build doctor alone is enough for the portfolio demo.