claude-skills/.wiki/concepts/active-platform-eval-design.md

title, type, tags, updated

title	type	tags	updated
Design: active-platform eval + tuning	concept	active-platform eval skill-creator tuning	2026-05-05

Design: active-platform eval + tuning

Tune the active-platform skill (description + body) using skill-creator's run_loop.py for description optimization, executed in parallel with a manual body sweep. Replaces the placeholder "wait for 5 real signals" approach with synthetic, balanced signals across Windows / Linux / macOS. Absorbs the two ⚪ tasks [active-platform-tuning] and [active-platform-eval] into a single workstream — they're inseparable in practice (eval is the tuning mechanism).

Why combine the two tasks

[active-platform-tuning] was originally framed as "tune description based on real-usage signals". [active-platform-eval] was "run the formal skill-creator eval loop, blocked by tuning gathering signals". The blocker assumed signals come from production — but a synthetic eval set covers the same surface area in one pass, with controlled cross-platform balance, and produces both the data and the tuned description in the same run. Splitting them was an artefact of the original "wait for misfires" mental model. We're not waiting any longer.

Scope

In scope:

1. Build a 20-query trigger eval set covering Windows, Linux, macOS, and ambiguous shell-context cases, plus near-miss negatives.
2. Run skill-creator/scripts/run_loop.py (max 5 iterations, train/test split) against the current active-platform/SKILL.md description, capturing best_description.
3. In parallel with the loop, manually sweep the SKILL.md body for known weak spots (WSL handling, BSD/macOS divergences coverage, ambiguity policy clarity).
4. Apply best_description to the frontmatter; apply manual body edits.
5. Bump version: 1.0.0 → 1.1.0 (MINOR — improved triggering + expanded body coverage; behavior compatible).
6. Rebuild dist/active-platform.skill, reinstall to ~/.claude/skills/active-platform/.
7. Document outcome (eval set, before/after scores, body diff summary) in .wiki/concepts/active-platform-eval.md (separate from this design doc — this is what we're going to do; that's what we did).

Out of scope:

• Description optimization for any other skill (this is just active-platform).
• Renaming the skill or restructuring the trigger phrase table.
• Changing the cross-platform-docs convention (that lives in the skill body and is fine).
• Building install.ps1 or any cross-platform install work — that's [install-ps1], downstream of this task.

Eval set composition (20 queries)

10 should-trigger queries, distributed:

Bucket	Count	Examples (sketch — final list lives in .tasks/active-platform-eval/eval-set.json)
Windows-explicit	≥3	"мы на винде, дай команду установки X", "I'm on Windows, give me the README quick-start", "сейчас под виндой, как поставить Y через scoop"
Linux-explicit	≥3	"we're on a Linux box, write a one-liner для systemd unit", "I'm on Ubuntu prod, как настроить cron", "переключись на линукс, нужен install для apt"
macOS-explicit	≥3	"я на маке, как поставить через brew", "I'm on a Mac, how do I configure zsh prompt", "switch to macOS, дай команду для plist"
Implicit shell context	rest	"give me README quick-start for a CLI tool", "напиши one-liner установки", "how do I run this in a fresh terminal" — should still trigger because output is shell-bound, even without OS-trigger word

10 should-not-trigger queries — near-misses, not obvious negatives:

Bucket	Count	Sketch
Library API questions	3	"как использовать subprocess.run в Python", "what does os.path.join do" — touches shell-adjacent territory but is code-question, not produce-commands-for-user
Architecture / design talk	2	"should I use Redis or Postgres for X", "explain microservices vs monolith" — no command output
README content questions (non-commands)	2	"what should the description section of a README contain", "how do I write a good API doc" — README meta, not quick-start
Per-question override tests	2	"how would this look on the prod box, which is Ubuntu" — should trigger (per-question), but must not flip the session-wide default away from current. The skill body addresses this; eval should test the description doesn't over-trigger and switch sessions on stray "on Ubuntu" mentions. Mark as should-not-trigger session-switch; ambiguous for skill activation — TBD during eval set finalization.
Genuinely unrelated baseline	1	"explain the halting problem" — sanity check that the negative bucket isn't all subtle-near-misses

Final eval set committed at .tasks/active-platform-eval/eval-set.json before the run starts (so the eval is reproducible from git).

Workflow (parallel)

┌─ MAIN AGENT ─────────────────────────────────┐
│                                              │
│  1. Snapshot active-platform skill           │
│  2. Build + commit eval set                  │
│  3. Launch run_loop.py in background ────────┼─→ ┌─ BACKGROUND ──┐
│                                              │   │ run_loop.py   │
│  4. (parallel) manual body sweep:            │   │ ~5-10 min     │
│     — re-read SKILL.md fresh                 │   │ 5 iter × 20q  │
│     — note WSL gap, BSD-coreutils gap,       │   │ × 3 reps,     │
│       ambiguity policy clarity               │   │ train/test    │
│     — draft revised body sections            │   │ split         │
│                                              │   └───────────────┘
│  5. ← run_loop completes → read              │            │
│     best_description from output JSON        │ ←──────────┘
│                                              │
│  6. Apply best_description to frontmatter   │
│  7. Apply body sweep edits                   │
│  8. Bump version 1.0.0 → 1.1.0               │
│  9. Rebuild dist/, reinstall                 │
│ 10. Write .wiki/concepts/active-platform-    │
│     eval.md (results + before/after)         │
│ 11. Update wiki index + log                  │
│ 12. Update STATUS.md (both ⚪ → 🟢)          │
│ 13. Commit, ask before push                  │
│                                              │
└──────────────────────────────────────────────┘

The two halves are independent: run_loop.py only touches the description field via its own iteration loop; manual body edits don't conflict because they target the body. Both converge in step 6-7 where I apply the diffs in sequence.

Tooling concerns

run_loop.py location: <skill-creator-cache>/skills/skill-creator/scripts/run_loop.py. Need to invoke python -m scripts.run_loop from that directory.

Model ID for the loop: claude-opus-4-7 (current session model — the loop's triggering test should match the model the user actually experiences).

claude CLI dependency: run_loop.py shells out to claude -p to test triggering. Need to confirm the claude CLI is on PATH before kickoff. If missing, fallback to manual single-pass description rewrite.

Workspace path: .tasks/active-platform-eval/ — gitignore the per-iteration subdirectories (transient outputs, benchmarks, HTML reports), commit only eval-set.json and the final summary in .wiki/concepts/active-platform-eval.md. Add a .gitignore line scoped to that workspace.

Versioning + manifest

version: 1.0.0 → 1.1.0 — MINOR per project-discipline Rule 3:

• Description rewrite = improved triggering, behavior compatible.
• Body additions (WSL clarity, BSD/macOS expansion, ambiguity policy refinement) = added capability, no removed coverage.

bootstrap-manifest.md (in projects bootstrapped from this repo) lists active-platform: 1.0.0 — bumping doesn't propagate retroactively, but new bootstraps after this commit will pick up 1.1.0 automatically via the manifest-write step.

Spec self-review

• ☑ No "TBD" — except one explicit ambiguity flag in the eval set (per-question override category) that's resolved during eval set finalization, not deferred.
• ☑ Internal consistency: Workflow step 6 applies best_description, step 7 applies body edits, step 8 bumps version — all in sequence, no contradiction.
• ☑ Scope: focused on active-platform. [install-ps1] mentioned only as downstream sequencing; explicitly out of scope.
• ☑ Ambiguity: should-not-trigger for "per-question override" cases is the one tricky spot. Resolution: those queries should trigger the skill (the body explicitly handles per-question), but should NOT flip the session-wide default. Eval can only measure trigger-or-not, not session-state mutation — so mark them should_trigger: true and rely on the body content to handle the no-flip behavior. Documented in eval set comments.

Connection downstream

After this task:

• [install-ps1] becomes the obvious next item (cross-platform install scripts pair with the now-better-triggered active-platform skill).
• active-platform-tuning-v2 may emerge later if real-world misfires surface gaps the synthetic eval missed; tracked as a fresh task at that time, not now.

See also

• active-platform-decision — original decision to make active-platform a skill (not memory); open questions list (WSL, macOS coverage) is what this eval surfaces and resolves
• skill-versioning — semver discipline that drives the 1.0.0 → 1.1.0 bump
• bootstrap-skill-deps-check — active-platform is one of the 9 canonical fulfillers in the bootstrap dependency map; description tuning here doesn't change its kind: skill classification