Before an AI change goes live, I ask what breaks if it is wrong

AI agents write most of the code behind this blog, including the security settings. That is a job where sounding sure and being right come apart. An agent writes a security rule in the same calm tone whether it is right or wrong, and some rules, wrong by one line, take the whole site down for every visitor.

So I sort what an agent hands me into two piles: changes that fail small, and changes that fail everything. They do not get the same treatment.

In July an agent added a batch of security settings to this site. Most were the fail-small kind:

• Stop other sites from showing these pages inside their own.
• Stop the browser from guessing what kind of content a page holds.
• Force every visit onto a secure connection.

Get one of those wrong and a page is a little less protected. It still loads.

One setting was different. A Content Security Policy is a list that tells the browser exactly where a page may load scripts and styles from. Anything not on the list gets refused. That is great against a hijacked page. It is also a rule that, written one entry too tight, refuses your own site’s code, and every visitor gets a broken page. You usually find out from an email, not a test.

So that one shipped in watch mode. The browser reads the rule, and anything it would have blocked gets written to a log instead of blocked. The site runs exactly as before. I get a list of everything the rule would have broken, and no visitor pays for the mistake.

Small honest number: it is still in watch mode today, three months later. It only moves to enforcing after a clean log on every kind of page the site serves, and I have not done that check yet. Watch mode costs me nothing while it waits. Enforcing it early could cost me the whole site.

The habit is one question, asked before anything an agent wrote goes live: if this is wrong, what breaks?

• If the answer is “one page looks off,” ship it and fix it later.
• If the answer is “everything, for everyone,” make it run in a mode that can only report the problem first.

Most tools that can break everything have a mode like that: a dry run, a preview, a test send to yourself. Use it every time the answer is everything.