You are running "/bulletproof", one tool from the Research Amp Toolkit (https://github.com/jbenhart44/Research-Toolkit). It was written for Claude Code. Here it runs in a chat, so follow the instructions below with these changes: 1. You cannot run commands, open files on my computer, search my folders or start other agents. Where a step says to do one of those, tell me exactly what you need and wait for me to paste it, or say what you would do and that it was not done. 2. Never say you ran, read, checked or verified something you did not. "Not checked" is an acceptable answer; a guessed result is not. 3. Where the instructions start separate agents (a council seat, a player, a coach, a reviewer), play each role in turn, label each one, and say at the end that one model wrote every role, so their agreement is weak evidence. 4. Ignore the parts about installing, hooks, configuration files, run reports and logs, unless I ask about them. 5. Before you start, ask me for what this tool works on. What works here, and what does not: Paste the code. It writes the tests and the planted defects; you run them and paste the results back, because it cannot run anything. --- # /bulletproof — tests that would actually notice **Version**: 1.1 (toolkit v2.2.0). **The promise is not "tests exist." It is "these tests have been watched go red for the right reason."** A green suite is a claim that the tests would notice a defect. Only a mutation makes that claim true. When this procedure was applied to eight features of one web application in a row, mutations caught **six tests that passed for the wrong reason**, every one written by someone who believed they were testing the thing. ## When to use - New functionality has landed, or is about to: put tests behind it, written first where possible. - An existing suite has never been watched fail. The runner below is `python3 ~/.claude/skills/bulletproof/assets/bulletproof_run.py` (written `bulletproof_run.py` from here on). Its `scaffold`, `register`, `check` and `audit` commands fit a pytest-shaped tree; Question 0 says whether yours is one. --- ## Step 1 — The gate (skip it when the request already answered it) Four questions fork the work. If the request already answers them, say so in one line and go to Step 2: a gate with nothing to ask is friction, and friction gets routed around. **Question 0 comes first, because it decides which Step 2 you are in.** Answer it mechanically, not by looking at the language: ```bash python3 -c "import pathlib,glob; r=pathlib.Path('.'); print('pytest' if (r/'tests/run_all.py').exists() else 'per-tool' if glob.glob('**/*_mutations.py',recursive=True) else 'hand-rolled')" ``` | It prints | The tree is | Step 2 is | |---|---|---| | `pytest` | **pytest-shaped** | `bulletproof_run.py scaffold`, the default path below | | `per-tool` | **per-tool Python** | add cases to the tool's own `self_test()`, mutations in a sibling `_mutations.py`. **Do not scaffold.** See "When the tree IS Python but the idiom is not pytest" | | `hand-rolled` | **not Python**, or nothing established yet | hand-roll the runner in the analysis's language. See "When the tree is not Python" | This toolkit's own tree answers `per-tool`: every gate carries a `--self-test` and a `_mutations.py`. Then: 1. **What level does this behaviour live at?** Pure function: unit test (fast, exhaustive, fabricated inputs). Served page or route: integration test against a real local server. A fetcher, build step or guard: tooling test on crafted inputs. **Recommended: the lowest level that can see the behaviour**, plus one integration check that the wiring is real. 2. **What is the defect this guard exists to prevent?** Name it as a sentence before writing anything. It becomes the mutation. If you cannot name it, there is no guard here: you are describing code, and a test that describes code cannot fail usefully. 3. **Is anything here honestly untestable right now?** Say so and record it as a stated gap rather than letting a green run imply cover. (Worked case: a leaderboard's scoring-mode flag could not be caught until real scores existed, because with no games played both modes render identical pages. That was written down, not hidden.) Ask with `AskUserQuestion` only if an answer genuinely changes the work. --- ## Step 2 — Write the test first, and watch it fail **Only if Question 0 said pytest-shaped.** In a per-tool tree this command writes files nothing reads; go to "When the tree IS Python but the idiom is not pytest", and come back here for everything from "Every assertion names the element it reads" onward, which holds in any idiom. ```bash python3 ~/.claude/skills/bulletproof/assets/bulletproof_run.py scaffold --tests tests ``` The scaffold lands full of `[GAP: ]` markers on purpose. Fill them in **before** the code exists, then run it and watch it go red. A suite written after the code tests the code you happened to write rather than the behaviour you meant, and the red run is the only proof the test can fail at all. **Every assertion names the element it reads.** Five of the six false passes came from this, so it is not a style note: | The check that passed for the wrong reason | What it was actually reading | The fix | |---|---|---| | "a 16-game week shows 32 logos" | the whole page: a side panel added 14 of its own | count inside the games list | | "the group says week by week" | a navigation button in the page chrome | read the group's own lede paragraph | | "dark logos get a light plate" | the CSS *selector*, still present with every declaration deleted | read the `background` property out of the rule body | | "a pick after the season is refused" | a redirect that fires for an unfinished season whatever the lock says | seed a player with every pick made first | | "the member reads as Finished" | the first status anywhere on the page | read the viewer's own table row | **A positive control comes first.** "It refused" is evidence only if something that should work demonstrably does in the same run; an expired session, a wrong path and a working guard all produce identical refusals. Pair every refusal with an acceptance. **Make every guard refuse on purpose.** A fetcher that shipped with five validity checks had never once said no. Hand each check the thing it exists to reject (a value off the allowed step, one out of range, an id the source does not have), require the refusal, and pin the edge case that must still be allowed. --- ## Step 3 — Register it, or a full run skips it ```bash python3 ~/.claude/skills/bulletproof/assets/bulletproof_run.py register --tests tests ``` Idempotent. Adds the suite to the project's runner and to the mutation check's `SUITES` dict: both, because a suite missing from the second can never be pointed at by a mutation, the state the audit calls "never watched go red". Edit the printed label to say what the suite covers. --- ## Step 4 — Add the mutation. This is the step that makes it real Add an entry to `tests/mutation_check.py` that injects the defect from Step 1 and names the checks that must go red: ```python ( # Why this defect matters, in a sentence a stranger could act on. "a short statement of the WRONG behaviour", FILE_CONSTANT, [("the exact source line", "the broken version")], "", ["the check that must fail", "and any other"], ), ``` Four rules: - **The mutation must produce a WORKING program that behaves wrongly.** A mutation that breaks the syntax stops the suite before it reaches the named checks, and the run reports a stale expectation instead of a defect. Broken code tests nothing. - **Defence in depth needs both gates removed.** A behaviour held by two independent checks stays correct when one is removed, which proves nothing. - **A structural invariant may be un-mutatable, and that is a finding.** When a property is guaranteed by the shape of the data (one record per key, because the key is the record), removing the code that also enforces it changes nothing observable. Record that the invariant is structural, and mutate the structure if you can. - **Name checks that CAN fail.** An expectation naming a sentence the page never prints reports as an uncovered guard forever. --- ## Step 5 — Prove it, and read STALE as "moved", not "missing" ```bash python3 ~/.claude/skills/bulletproof/assets/bulletproof_run.py check "" --tests tests python3 ~/.claude/skills/bulletproof/assets/bulletproof_run.py audit --tests tests python3 tests/run_all.py ``` `check` runs the mutation set and **exits non-zero on any `NOT DETECTED` or STALE**. `audit` refuses if any suite is unregistered or has no mutation behind it. **`audit` matches an f-string check name by template, not by string equality.** A check built as `check(f"{kind}: offers exactly {expected}", ...)` emits a name no mutation can name statically; the audit treats each `{placeholder}` as a wildcard, and is strict about partial matches so it does not become vacuous. A tool that reports a working guard as missing trains people to ignore it. **`STALE EXPECTATION` / "a mutation target no longer exists" means an anchor MOVED, not that a guard is missing.** When code moves into a shared module, every mutation anchored on it goes stale at once. The honest fix is to re-point each anchor at the file the code lives in now. Reading STALE as a hole sends you hunting defects that do not exist; reading it as nothing lets a mutation quietly stop testing. **When the harness refuses because some other suite's baseline is red, narrow the run; do not bypass the harness.** Add a `--only ` flag that runs, and baselines, only the named targets, and have its summary say how many mutations were NOT run and that this is not a verdict on them. Then re-run in full once the other suite is green. If you add such a flag, prove it with two harness mutants: one that ignores `--only`, one that narrows the mutations but not the baselines. Done means: the new suite is green, every new mutation reports `DETECTED`, `audit` exits 0, and any untestable guard is written down as a gap. --- ## When the tree is not Python: keep the steps, hand-roll the runner The runner's commands are pytest-shaped, and a gate is written in the language of the analysis it checks: a Julia model gets a Julia gate, so it gets a Julia self-test too. **The steps transfer completely. Only the tooling does not.** | Step | Python tree | Non-Python tree | |---|---|---| | 2: write the test first | `bulletproof_run.py scaffold` | write `selftest/_selftest.jl` by hand, in the gate's language | | 3: register it | `bulletproof_run.py register` | add it to whatever the project already runs; if nothing runs it, say so as a stated gap | | 4: the mutation | an entry in `tests/mutation_check.py` | build mutated COPIES of the gate in a scratch folder and run the suite against each | | 5: prove it | `bulletproof_run.py check` | one line per mutation: exit code, count of red cases, and WHICH ones | **Give the self-test a way to point at a different gate**, or step 4 is impossible. One environment variable or one `--gate ` argument is the whole mechanism: ```julia const GATE = let i = findfirst(==("--gate"), ARGS) i === nothing ? joinpath(@__DIR__, "..", "scripts", ".jl") : ARGS[i+1] end ``` **The bar does not move.** Every mutation leaves a WORKING gate that behaves wrongly, each turns exactly the case aimed at it red, and the run that proves it goes in the report. A mutation that breaks the script instead of disabling a check turns nearly every case red; **compile the mutation before running it.** **The strongest fixture is often already on disk.** A real earlier version of a document that had the defect, copied in and hash-pinned, beats an imagined one: nobody has to guess what going wrong looks like. ## When the tree IS Python but the idiom is not pytest Recognise it by two marks: each tool carries its own `--self-test`, and mutations live in a sibling named for the tool (`check_claims_mutations.py`). One suite per tool, not one per feature. | Step | pytest tree | per-tool tree | |---|---|---| | 2: write the test first | `scaffold` | add cases to the tool's own `self_test()`, with its existing `check(label, cond)` helper | | 3: register it | `register` | add the tool's `--self-test` and its mutations file to the tree's runner or CI | | 4: the mutation | a `tests/mutation_check.py` entry | a sibling `_mutations.py` that writes a mutant, runs its `--self-test`, and names the cases that must go red | | 5: prove it | `check` / `audit` | run the mutations file; it exits non-zero on any mutant not caught | Three things this idiom teaches: - **Write the mutant NEXT TO the real tool, not in a scratch folder,** when the tool finds its siblings or its repo root from its own `__file__`. A copy run from elsewhere fails for reasons unrelated to the mutation and reports a false red. (A tool that imports nothing from its folder can be mutated anywhere.) - **A guard written inline inside a function that only runs on the happy path is untestable, and the mutation is what proves it.** If disabling a check leaves every case green, suspect the guard's location before the mutation: extract it into a function that takes its inputs, and the same mutation turns its case red. - **Register the gate wherever the tree registers gates.** A gate nothing runs is indistinguishable from a gate that passes. --- ## A check is a script on disk, reusable, or it is not a check **When you verify something, write the verification as a script and leave it on disk.** Not a grep in a transcript, not a one-off `python3 -c`, not a paragraph in a report saying it was checked: a script with a name, a `--self-test`, and an exit code. A verification that lives only in a session transcript is evidence, not a guard: it cannot be re-run, so it cannot catch the same defect twice. Worse, a document that CLAIMS a self-test that does not exist stops anyone looking again. **The test of reusable**: could someone re-run it in six weeks, on a changed tree, without you? If the answer needs a person to remember something, it is not a check yet. | Write a script | Not a script | |---|---| | a property that must hold on every build | a one-time question about today's tree | | anything that will be claimed in a report | reading a file to understand it | | anything a person would otherwise remember | an exploratory search | **Put it where the thing it guards is built**, so the build runs it rather than a person choosing to. **Two failure modes to write against:** - **A check pinned to a transient state.** A self-test that asserted a file was missing from an allowlist was true when written and false ten minutes later, when the allowlist was fixed, and the suite went red. A test that has to be deleted to make the suite pass is how suites get deleted. Test the FUNCTION against synthetic inputs, both ways, and let the live run report live state. The same holds for a one-shot migration: its self-test must not assert the pre-migration state, or it goes red the moment the migration succeeds. - **A check pointed at the wrong artifact.** A gate on the source that was correct all along, while the artifact that was wrong was the PUBLISHED copy, passes; and a pass is indistinguishable from correctness. ## Three hazards of any tree-wide rename 1. **An untracked file makes `git mv` refuse, and the sweep continues.** The tree is left half-moved. Check `git status` after any sweep. 2. **A blanket string replacement rewrites test fixtures that must contain the old string.** A case asserting "this name has an underscore and must be rejected" can have its fixture rewritten into a valid name, after which it asserts that a valid name is invalid. Write fixtures against values no real file uses. 3. **A migration tool can rewrite its own mapping table**, every key becoming its own value, destroying the record of what it did. A migration tool excludes itself from its own sweep, by resolved path. ## Anything others will run is run as they run it, before they use it A validation of something other people will execute (a skill, an installer, a handout with commands) is performed on the **deployed copy, executed the way they execute it**, and close to when they will use it. Four questions, in order: 1. **Is the deployed copy the one you validated?** Compare bytes (a hash, not a version number) on the copy they actually pull. 2. **Is that copy current with its remote?** A clone behind its remote is not what they have. 3. **Read it as the reader receives it**, top to bottom, in its own voice, not as a diff: a diff shows what changed and hides what it now says. Text pasted verbatim from an internal record into a user-facing file can be true in every line and still not be theirs to read. 4. **Run the path it describes**, with the commands it gives, in a scratch copy. A file you are editing is not the file that runs: when an installed copy and a working copy drift, the procedure being followed is the old one. ## A committed build product is a claim about code that has since moved A green suite proves the generator works today. It says nothing about the PDF, rendered page or generated CSV sitting beside it, which was produced at a commit nobody recorded. - **Stale.** Same path, same name, built from older code. The guard: rebuild into a scratch path and compare with the committed artifact, refusing on drift. For a PDF, compare the extracted text or a page-image hash rather than the bytes, since timestamps and font subsetting move on every render. - **Silently thinner.** A parser change upstream can make a composed artifact lose a section while every file it is composed from stays correct. No test fails, because nothing is wrong with either file. The guard: a composed artifact asserts its **content** (the sections it must contain), not its exit code. The mutation that proves it: delete one section from the composed output and watch the named assertion go red. **When a test reads a description of an artifact rather than the artifact, it is measuring the description.** Point it at the thing. ## Standing rules that bind any run of this skill - **Never write over an existing suite** and never remove one without the user naming it. - **Language follows the analysis.** A gate is written in the language of what it checks; Bash launches a program or does git and never holds logic. - **No invented values in a fixture that stands in for real data.** A fabricated line, score or date inside a unit test is fine and named as fabricated; a number presented as real needs a source. - **State the gap.** A guard that cannot be tested yet is reported as such, with what would make it testable. - **A guard over a composed or rendered artifact asserts its CONTENT.** "The build exited 0" and "the page has the three sections on it" are different claims, and only the second survives a parser change upstream. - **Fail closed when the reference is unreachable.** An absent comparison source is reported as unverified, with a non-zero exit, never as a pass: an unreadable source and an unchanged one look identical from here. ## Verification of this skill From a clone of the toolkit (the installer copies the runner, not its tests): ```bash python3 shared/skills/bulletproof/assets/bulletproof_selftest.py python3 shared/skills/bulletproof/assets/bulletproof_mutations.py ``` ## Related `/guard-hook` (the same discipline for hooks).