From 2ba698185bd9f3b19ac8fbd338a2eba1b108173f Mon Sep 17 00:00:00 2001 From: vitya Date: Thu, 7 May 2026 08:24:05 +0300 Subject: [PATCH] =?UTF-8?q?docs(tdd-criteria):=20rule=204=20=E2=80=94=20te?= =?UTF-8?q?st-immutability=20defence=20(was=20X;=20is=20Y=20marker)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 --- .wiki/concepts/tdd-criteria-design.md | 46 +++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/.wiki/concepts/tdd-criteria-design.md b/.wiki/concepts/tdd-criteria-design.md index 3bebfce..fca4c02 100644 --- a/.wiki/concepts/tdd-criteria-design.md +++ b/.wiki/concepts/tdd-criteria-design.md @@ -4,6 +4,7 @@ source: .meeting-room/.archive/2026-05-07-tdd-criteria.md status: promoted type: design title: tdd-criteria-design +amended: "2026-05-07: added test-immutability defence (Anti-loophole rule 4) after user noted symmetric vandalism risk on tests" ingested_at: '2026-05-07T04:01:23.616Z' ingested_by: OpeItcLoc03@DESKTOP-NSEF0UK source_project: .meeting-room @@ -33,6 +34,17 @@ In a workflow that includes LLM coders (Claude, ChatGPT, future unknowns) and ro This reframes everything below: +### The contract is only as strong as the contract itself + +But there's a **second-order vandalism mode** that the bare contract argument doesn't cover: the agent doesn't delete the code, it rewrites the **test**. Test fails → agent changes the expected value, adds `.skip`, or deletes the test → test now passes → success reported. + +If the contract artefact (the test) is rewritable by the same agent that's failing to satisfy it, the invariant collapses back into an artefact. The defence requires **two layers**: + +1. **Code is defended by tests.** Ironclad rules 1-4 below. +2. **Tests are defended by process discipline.** Anti-loophole rule 4 below — append-only by default, modifications require literal-`was/is` marker in commit subject, test changes are separate commits from impl changes. + +Both layers are needed. Either alone leaves a path-of-least-resistance route to «success». + - **Ironclad** rules (TDD obligatory) — zones where recovery cost > defending cost. Signal mandatory. - **Permissive** carve-outs — **zones of accepted risk for agentic vandalism**, not «zones where TDD doesn't apply». You explicitly accept that an agent can delete-and-claim-success here, because recovery is cheap (eyeball the next render; throwaway by contract; one-shot already ran; wrapper trivial to reconstruct). @@ -85,9 +97,35 @@ Not «TDD doesn't apply». **«You accept that an agent can vandalise this witho ## Anti-loophole -- **Skip без категории не существует.** One of four explicit categories — not «other reasons». No marker = violation, regardless of perceived justification. -- **Spike survivor rule.** If spike code is merged to master → same merge-commit creates `[backfill-tests-]` task in `.tasks/STATUS.md`. Otherwise «spike» becomes «skipped tests forever». -- **Friction is the point.** `[skip-tdd: visual]` 50 раз подряд при `web-design-system` итерации раздражает — это и есть замысел. Friction = fence, not bug. If it becomes unbearable, re-evaluate after ≥2 weeks of usage, not before. +1. **Skip без категории не существует.** One of four explicit categories — not «other reasons». No marker = violation, regardless of perceived justification. +2. **Spike survivor rule.** If spike code is merged to master → same merge-commit creates `[backfill-tests-]` task in `.tasks/STATUS.md`. Otherwise «spike» becomes «skipped tests forever». +3. **Friction is the point.** `[skip-tdd: visual]` 50 раз подряд при `web-design-system` итерации раздражает — это и есть замысел. Friction = fence, not bug. If it becomes unbearable, re-evaluate after ≥2 weeks of usage, not before. +4. **Tests are append-only by default** (added 2026-05-07). New tests: free. **Modifying** an existing assertion, **deleting** a test, or **disabling** it (`it.skip`, `xit`, `@pytest.mark.skip`, `@Disabled`, etc.) requires both: + + **a)** A marker in commit subject: + ``` + [test-modify: : was ; is ; reason: ] + ``` + Where `` and `` are the **literal assertion expressions** before and after, not paraphrased. Example: + ``` + [test-modify: validates email format: was expect(isValid("a@b")).toBe(true); is expect(isValid("a@b.com")).toBe(true); reason: tightened spec to require TLD] + ``` + + **b)** Test changes go in a **separate commit** from any impl changes. A single commit must not modify both `*.test.*` and `src/*` files (or their project-equivalents). This forces an audit-able split — `git log --grep '\[test-modify'` shows every test rewrite cleanly. + + **Why literal `was/is`, not free-form reason:** an agent forced to write the literal assertion publishes exactly what they're rewriting. If `42` was the correct expectation and they changed it to `43` to make a buggy fix pass, the literal `was 42; is 43` line in `git log` identifies the culprit. A reason like «updated to match new behaviour» hides everything — agents will use it whenever it is allowed. + + **Bright-line check** (for an optional pre-commit hook, see follow-up task `tdd-criteria-precommit-hook`): + - `git diff --cached` includes a removed `expect(...)` / `assert(...)` / `assertThat(...)` line, OR + - changes the arguments of an existing assertion call, OR + - adds `.skip`, `xit`, `@skip`, `@Disabled`, etc. annotation, OR + - deletes a test file or `it(...)` / `test(...)` / `def test_*` definition + + AND the commit subject does not contain `[test-modify: ...]` matching the format above → block. + + AND `git diff --cached --name-only` includes both test-pattern and impl-pattern files → block (require split). + +5. **Don't apply rule 4 retroactively** to tests written before the rule was adopted. The rule applies to test changes made after the project's CLAUDE.md picks up `follow tdd-criteria`. Existing test bodies aren't grandfathered into requiring `was/is` for a one-time rewrite. ## What's excluded as not bright-line @@ -97,6 +135,7 @@ These tempting formulations were rejected: - ~~«When time is short»~~ — time is never abundant. - ~~«When the logic is obvious»~~ — it seems obvious until the first bug. - ~~«When the code is temporary»~~ — there's no such thing; temporary code becomes permanent. +- ~~«Tests should not be modified casually»~~ — paraphrasable, agents will modify casually and call it «refactor». Replaced by Anti-loophole rule 4 with literal-evidence requirement. All require judgment at decision time → automatically become loopholes. The four Permissive categories are bound to **observable properties** (file extension, marker in commit subject, directory, line count) — not mood. @@ -122,6 +161,7 @@ Project `CLAUDE.md` files may **extend** Permissive (e.g. `karu` is a pure-CSS p - **Supportive evidence is n=1.** The observable contrast is `books` (TDD applied selectively, features ship) vs `modules-db`/`pilonuxt` (TDD not applied, projects buksuyut). `books` could be more productive for reasons unrelated to TDD (codebase maturity, author experience, task type). The direction-of-effect agrees with first principles, so inversion is unlikely, but the magnitude is uncertain. - **Friction in UI iterations.** `[skip-tdd: visual]` repeated 50× during a design-system rework is annoying. Intentional. Don't relax before ≥2 weeks of usage. +- **The literal-`was/is` requirement is verbose** for a renamed test or trivial typo fix. The verbosity is the point — an agent that genuinely fixed a typo writes the same assertion twice with one character changed; an agent that rewrote a failing test writes obviously different assertions. Reading `git log --grep '\[test-modify'` shows the difference at a glance. - **«Don't codify at all» was rejected.** Precisely «not codified» is what produced the situation in `books` where, after the author left, his TDD criterion can't be reconstructed — it lived only in his head and his commits. A departed contributor is an argument *for* codification, not *against*. - **The anti-vandalism argument is uncomfortable.** It says: «I don't trust agents not to delete my code». That's the position. If it changes, this rationale changes. Until then, this is what's load-bearing.