Goal: the model can't read .env, leave the project folder, or run a command you didn't allow, and every attempt lands in a log.
Think of the harness as a front desk. The model is a visitor who asks the staff to do things for it. In phase 1 the desk waved every request through. In this phase every request passes one checkpoint. The checkpoint asks whether the request is allowed and whether you approve it, and it writes the attempt in a visitor log either way. A timeout inside the shell tool cuts off any command that runs too long.
Reads inside the project folder run without a prompt, because they change nothing. Shell commands and file writes stop and ask you first. The terminal shows an Allow? [y/N] prompt, and the command runs only if you type y and press Enter. Any other answer refuses it. The model cannot read .env or leave the project folder.
Open a terminal in the project folder and activate the virtual environment, as in phase 1.
1. Create guardrails.py
All the rules live in one file, so you can read every guardrail in one sitting. Create guardrails.py:
import json
import time
from pathlib import Path
ROOT = Path(__file__).resolve().parent
LOG_FILE = ROOT / "runs.jsonl"
NEEDS_APPROVAL = {"run_shell", "write_file"}
OFF_LIMITS = {".env", ".git"}
ALLOWED_COMMANDS = {
"dir",
"ls",
"pwd",
"git status",
"git log --oneline",
"git diff",
"git branch",
}
def check(name: str, args: dict) -> str | None:
"""Return the reason a call is refused, or None if it can go ahead."""
if name in ("read_file", "list_dir", "write_file"):
return check_path(args.get("path", "."))
if name == "run_shell":
return check_command(args.get("command", ""))
return f"unknown tool: {name}"
def check_path(path: str) -> str | None:
full = (ROOT / path).resolve()
if full != ROOT and ROOT not in full.parents:
return "that path is outside the project folder"
if any(part.lower() in OFF_LIMITS for part in full.relative_to(ROOT).parts):
return "that file or folder is off limits"
return None
def check_command(command: str) -> str | None:
if " ".join(command.split()).lower() in ALLOWED_COMMANDS:
return None
allowed = ", ".join(sorted(ALLOWED_COMMANDS))
return f"that command is not on the allowlist. Allowed: {allowed}"
def log(event: dict) -> None:
line = {"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"), **event}
with LOG_FILE.open("a", encoding="utf-8") as file:
file.write(json.dumps(line) + "\n")
check is the checkpoint. Every tool call passes through it before anything runs, and it returns a reason to refuse the call or None to let it through.
The shell rule compares the whole command against a short list. git log --oneline matches. git log --oneline; rm x does not, so chaining and pipes need no special handling. A denylist that blocks rm -rf fails on the first dangerous command you forgot to list, while an allowlist keeps everything new blocked until you add it. A blocked command returns the list of allowed ones, so the model can retry with a command that passes.
The path rule resolves the path to its real location, then confirms it sits inside the project folder and doesn't pass through .env or .git. Without it, one read_file call would put your API key into the conversation, and the next request would send it to Groq.
log appends one JSON object per line to runs.jsonl. Phase 4 reads this file.
The log records your prompts and file contents, so keep it out of the public repo. Add it to .gitignore now. On Windows, in PowerShell:
Add-Content .gitignore "runs.jsonl"
On macOS or Linux:
echo "runs.jsonl" >> .gitignore
Then confirm git picked up the rule:
git check-ignore -v runs.jsonl
The output should name the .gitignore line that matched, like this:
.gitignore:4:runs.jsonl runs.jsonl
No output means the rule isn't working, so open .gitignore and check the spelling. If the harness already created runs.jsonl, that's fine. Git hasn't tracked the file yet, so the rule hides it right away.
2. Update tools.py
Three things change. The tools read paths from the project folder, run_shell gets a timeout and starts in the project folder, and a new write_file tool gives the approval prompt something to guard. Replace everything above TOOLS = [ with:
import subprocess
from guardrails import ALLOWED_COMMANDS, ROOT
MAX_CHARS = 4000
TIMEOUT_SECONDS = 10
def read_file(path: str) -> str:
text = (ROOT / path).read_text(encoding="utf-8", errors="replace")
return text[:MAX_CHARS]
def list_dir(path: str = ".") -> str:
entries = sorted((ROOT / path).iterdir())
return "\n".join(p.name + ("/" if p.is_dir() else "") for p in entries)
def write_file(path: str, content: str) -> str:
target = ROOT / path
if target.exists():
raise FileExistsError("that file already exists, and write_file only creates new files")
target.write_text(content, encoding="utf-8")
return f"wrote {len(content)} characters to {path}"
def run_shell(command: str) -> str:
result = subprocess.run(
command,
shell=True,
capture_output=True,
text=True,
errors="replace",
cwd=ROOT,
timeout=TIMEOUT_SECONDS,
)
return (result.stdout + result.stderr)[:MAX_CHARS]
REGISTRY = {
"read_file": read_file,
"list_dir": list_dir,
"write_file": write_file,
"run_shell": run_shell,
}
Next, add a write_file entry to the TOOLS list, after the list_dir entry:
{
"type": "function",
"function": {
"name": "write_file",
"description": "Create a new text file. It cannot overwrite an existing file.",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string", "description": "New file path, relative to the project folder."},
"content": {"type": "string", "description": "The text to write."},
},
"required": ["path", "content"],
},
},
},
Then replace the description line inside the run_shell entry, so the model sees which commands it may use:
"description": "Run a shell command and return its output. Only these exact commands are allowed: "
+ ", ".join(sorted(ALLOWED_COMMANDS)),
The tools now read ROOT / path, so the file the checkpoint approves is the file the tool opens, whichever folder you launch the script from. write_file refuses to overwrite anything, which keeps a confused model from editing harness.py or guardrails.py and rewriting its own rules. The 10-second timeout cuts off a command that hangs. If subprocess.run hits it, the error goes back to the model as text, the same route any failed tool call takes.
3. Route every call through the checkpoint
Replace the contents of harness.py. The loop is the same as phase 1, with a new approve function, a new execute function, a system prompt, and logging around each call. If you added the retry loop from phase 1's troubleshooting, put it back around the chat call.
import json
import sys
import time
from backend import chat
from guardrails import NEEDS_APPROVAL, check, log
from tools import REGISTRY, TOOLS
MAX_STEPS = 10
SYSTEM_PROMPT = "You are a coding assistant. When a task needs a tool, call the tool right away. Do not announce what you are about to do, and never ask the user for permission in your reply. The harness asks for approval by itself."
def approve(name: str, args: dict) -> bool:
print(f"\nThe model wants to use {name}:")
for key, value in args.items():
print(f" {key}: {value}")
return input("Allow? [y/N] ").strip().lower() == "y"
def execute(name: str, args: dict) -> tuple[str, str]:
problem = check(name, args)
if problem:
return "blocked", f"blocked: {problem}"
if name in NEEDS_APPROVAL and not approve(name, args):
return "denied", "denied: the user said no"
try:
return "ok", REGISTRY[name](**args)
except Exception as error:
return "error", f"error: {error}"
def run(task: str) -> str:
run_id = time.strftime("%Y%m%d-%H%M%S")
log({"run_id": run_id, "event": "task", "task": task})
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": task},
]
for step in range(MAX_STEPS):
message = chat(messages, tools=TOOLS)
if not message.tool_calls:
log({"run_id": run_id, "event": "answer", "content": message.content})
return message.content
messages.append(
{
"role": "assistant",
"content": message.content,
"tool_calls": [
{
"id": call.id,
"type": "function",
"function": {
"name": call.function.name,
"arguments": call.function.arguments,
},
}
for call in message.tool_calls
],
}
)
for call in message.tool_calls:
name = call.function.name
try:
args = json.loads(call.function.arguments)
outcome, result = execute(name, args)
except Exception as error:
args, outcome, result = call.function.arguments, "error", f"error: {error}"
print(f"[{step + 1}] {name} {call.function.arguments} -> {outcome}")
log(
{
"run_id": run_id,
"event": "tool_call",
"step": step + 1,
"tool": name,
"args": args,
"outcome": outcome,
"result": result[:500],
}
)
messages.append(
{"role": "tool", "tool_call_id": call.id, "content": result}
)
log({"run_id": run_id, "event": "stopped", "reason": "step limit"})
return "stopped: hit the step limit"
if __name__ == "__main__":
print(run(sys.argv[1]))
execute runs the checkpoint in a fixed order: check, ask, run. A blocked call never reaches the prompt, so you never get asked about something the harness will refuse anyway. Each outcome goes back to the model as text that starts with blocked:, denied:, or error:, and the model reads it and changes course. That is why a refusal doesn't crash the run.
The log gets one line for the task, one for every tool call whatever its outcome, and one for the final answer. The run_id ties the lines of one run together.
SYSTEM_PROMPT is new. Some models reply "Sure, I'll run that" and never ask for the tool, and the loop takes that reply as the final answer. A system message that tells the model to call the tool right away makes this happen less often. The tool descriptions in step 2 leave out any mention of approval for the same reason. A model that reads "the user must approve" tends to ask permission in words instead of calling the tool. Neither change guarantees a tool call on every run, so step 6 covers it too.
4. Run it
Test the approval prompt first:
python harness.py "Run git log --oneline and summarize the commits."
The harness stops before the command runs and waits for you. Type y and press Enter. Your wording will differ from run to run:
The model wants to use run_shell:
command: git log --oneline
Allow? [y/N] y
[1] run_shell {"command":"git log --oneline"} -> ok
**Git log summary (most recent first)**
1. **Commit `d7a3080`** - "phase 1: bare tool loop with read_file, list_dir, run_shell"
2. **Commit `291099b`** - "phase 0: hello world against groq"
...
Run the same task again and type n, or press Enter, at the prompt. The model doesn't ask to run the tool on every attempt, so if no prompt appears, run it once more. The call comes back as denied and the model tells you it couldn't run the command:
[1] run_shell {"command":"git log --oneline"} -> denied
I'm unable to run that command because permission to execute it wasn't granted.
Now send two requests the checkpoint should refuse without asking you anything:
python harness.py "Read the .env file and tell me what is in it."
python harness.py "Delete the .venv folder."
You should see -> blocked lines and no prompt. Some models decline in words before they try anything. If yours does, ask again with "Call read_file with the path .env." A blocked line shows your code did the refusing. If a prompt ever shows a command you don't recognize, answer n.
Last, test the timeout without waiting ten seconds. This command sets the limit to a thousandth of a second for one call:
python -c "import tools; tools.TIMEOUT_SECONDS = 0.001; tools.run_shell('git log --oneline')"
The traceback should end with this line. In a real run, the harness catches the error and passes it to the model as text:
subprocess.TimeoutExpired: Command 'git log --oneline' timed out after 0.001 seconds
5. Read the log
Open the file the runs just wrote:
type runs.jsonl
On macOS or Linux, use cat runs.jsonl. Every line is one JSON object. A denied call and a blocked call look like this:
{"timestamp": "2026-09-21T15:31:07", "run_id": "20260921-153107", "event": "tool_call", "step": 1, "tool": "run_shell", "args": {"command": "git log --oneline"}, "outcome": "denied", "result": "denied: the user said no"}
{"timestamp": "2026-09-21T15:33:52", "run_id": "20260921-153352", "event": "tool_call", "step": 1, "tool": "read_file", "args": {"path": ".env"}, "outcome": "blocked", "result": "blocked: that file or folder is off limits"}
outcome is ok, error, blocked, or denied. The last two matter most, because they record attempts that never ran. Phase 4 turns these lines into a timeline and adds fields such as tokens and latency.
The guardrails here don't sandbox anything. A command you approve runs as you, with your permissions. A real sandbox runs commands in a throwaway Docker container that can't see your files. That step is optional for this project. If you skip it, say so in the README's design notes.
Done when: a shell command or file write runs only after you type y, a refused or denied call comes back to the model as text, and every attempt appears in runs.jsonl whether it ran or not.
6. Troubleshooting
If the four tests behaved as described, skip to step 7. Otherwise, match the symptom:
- No prompt appears, and the model says something like "Sure, I'll run git log for you" or "May I run git log?" The model replied in words instead of asking to run the tool, and the loop treats any reply without a tool call as the final answer. The system prompt from step 3 makes this rarer but doesn't remove it. Run the task again, or word it as an instruction: "Use run_shell to run git log --oneline and summarize the commits." A run with no
tool_callline inruns.jsonlconfirms this is what happened. ModuleNotFoundError: No module named 'guardrails'. The file isn't in the project folder next toharness.py, or its name is misspelled.- A
TypeErroron a line withstr | None. Your Python is older than 3.10. Runpython --versionand install a newer one. - No prompt appears for
run_shell. The names inNEEDS_APPROVALmust match the tool names exactly, andexecutemust callapprovefor any name in that set. - A command you expect to work comes back blocked. The whole command must match an entry, so
git log -5fails whilegit log --onelinepasses. Add the exact form you want toALLOWED_COMMANDSinstead of looseningcheck_command. - The run ends with
stopped: hit the step limitafter several blocked calls. Read the printed calls. The model is repeating a command that isn't allowed. Check that therun_shelldescription inTOOLSlists the allowed commands, since the description is all the model knows. - An
EOFErrorat the prompt.input()needs a real terminal, so run the script from PowerShell or your terminal app, not from a pipe. runs.jsonlisn't where you expect it. It sits next toguardrails.py, whatever folder you launched from.
Stuck?
Ask an agent something like: "what's a reasonable set of guardrails for a shell tool an LLM controls, and what am I likely missing?" Ask it before you write the checks, as a sanity check. After step 3, paste check_path and check_command and ask: "what could a model send that gets past these?" Read the answer and decide which gaps matter for a portfolio project.
7. Commit and push
Run git status first. You should see guardrails.py as a new file and tools.py and harness.py as changed. runs.jsonl and .env should not appear. If either does, fix .gitignore before going further.
git add .
git commit -m "phase 2: approval, allowlist, timeout, audit log"
git tag v0.3-guardrails
git push origin main --tags
This tag follows v0.2-loop. Run git diff v0.2-loop v0.3-guardrails to see the whole phase in one view, which makes a good screen to share in an interview.