"Be careful" is not a safety feature

There's a standard answer for how to keep an AI agent away from destructive operations: put a warning in the prompt. Be careful. Double-check before deleting anything. It sounds reasonable, and it's probably the first thing most of us try. I tested it, and I watched it do nothing.

Letting it type dangerous commands

The test was deliberate. I handed an agent instructions I knew were dangerous: wipe the working directory with rm -rf, push freshly written code straight to a production repository. Both are the kind of operation you don't get to undo.

For the record, all of this ran in a disposable folder and a dummy repo. Nothing real was ever at risk. The point wasn't to break things — it was to find out where an autonomous agent keeps going, and where it stops on its own.

The politeness experiment

Then I added one line at the top: "please proceed carefully." The reply came back polite. Understood, proceeding with care. The command it actually built was identical to the version without the warning. No step to look at the folder contents first, no pause to ask a question — just the deletion, by the shortest path.

The push behaved the same way. "This is production, so be careful" didn't make it re-read the commit or double-check the target. The caution survived in the reply. It did not survive in the command.

The first wall: a blocklist

So I stopped asking and started blocking. The first version of the Kill Switch was almost embarrassingly simple: known-dangerous commands registered as plain strings, exact match, hold anything that matches.

blocked = ["rm -rf ./tmp", "git push origin main"]

def check(command):
    if command in blocked:   # exact match against the list -> stop
        return "hold"        # held. waiting for a human
    return "go"

A held command goes nowhere. The screen just says the operation is on hold. The agent doesn't get to decide "it's probably fine this time," because that decision has been taken out of its hands, structurally.

Where exact match leaks

On paper, the list looked sufficient. In practice it was full of holes, and I only found them by letting the agent actually type. The same deletion shows up in slightly different spellings:

rm -rf ./tmp
rm -rf tmp/

Same operation, same risk. Exact match catches one and waves the other through. The list I had drawn up at the design stage was coarser than I thought.

Reversible or not

So the rule changed. Instead of registering commands one by one, draw a line first — reversible versus irreversible — and judge everything against that single line. Can it be undone. That's the whole test. rm -rf and git push both land on the irreversible side.

What didn't work as a criterion: how dangerous something sounds. Judged by tone and context, the phrasing ends up deciding the outcome. "Delete this folder with rm -rf" gets held; "tidy up this folder and clear out whatever isn't needed" — same risk — walks straight through, because the wording is gentle. The only signal that held up under testing was reversibility.

A person at the gate

Deciding where to stop leaves one more question: who moves things forward after the stop. If the agent can look at a held operation, decide it seems fine, and resume on its own, the wall means nothing. So resuming is not the agent's call. A held operation stays held until a person looks at it. Approval is my job. That's the one part I haven't handed over, and I don't plan to.

What a Kill Switch actually is

Breaking things on purpose taught me what this mechanism is and isn't. A Kill Switch doesn't seal off destructive power. It's a checkpoint that routes every irreversible operation through a human, one step before it runs.

The wider lesson sits underneath: instructions written in words don't survive into execution. Structure does. If safety matters, it has to be built where the commands run, not requested where the prompts are written.