---
title: "Design: active-platform eval + tuning"
type: concept
tags: [active-platform, eval, skill-creator, tuning]
updated: 2026-05-05
---

# Design: `active-platform` eval + tuning

> Tune the `active-platform` skill (description + body) using skill-creator's `run_loop.py` for description optimization, executed in parallel with a manual body sweep. Replaces the placeholder "wait for 5 real signals" approach with synthetic, balanced signals across Windows / Linux / macOS. Absorbs the two ⚪ tasks `[active-platform-tuning]` and `[active-platform-eval]` into a single workstream — they're inseparable in practice (eval *is* the tuning mechanism).

## Why combine the two tasks

`[active-platform-tuning]` was originally framed as "tune description based on real-usage signals". `[active-platform-eval]` was "run the formal skill-creator eval loop, blocked by tuning gathering signals". The blocker assumed signals come from production — but a synthetic eval set covers the same surface area in one pass, with controlled cross-platform balance, and produces both the data *and* the tuned description in the same run. Splitting them was an artefact of the original "wait for misfires" mental model. We're not waiting any longer.

## Scope

**In scope:**
1. Build a 20-query trigger eval set covering Windows, Linux, macOS, and ambiguous shell-context cases, plus near-miss negatives.
2. Run `skill-creator/scripts/run_loop.py` (max 5 iterations, train/test split) against the current `active-platform/SKILL.md` description, capturing `best_description`.
3. In parallel with the loop, manually sweep the SKILL.md body for known weak spots (WSL handling, BSD/macOS divergences coverage, ambiguity policy clarity).
4. Apply `best_description` to the frontmatter; apply manual body edits.
5. Bump `version: 1.0.0 → 1.1.0` (MINOR — improved triggering + expanded body coverage; behavior compatible).
6. Rebuild `dist/active-platform.skill`, reinstall to `~/.claude/skills/active-platform/`.
7. Document outcome (eval set, before/after scores, body diff summary) in `.wiki/concepts/active-platform-eval.md` (separate from this design doc — this is *what we're going to do*; that's *what we did*).

**Out of scope:**
- Description optimization for any other skill (this is just `active-platform`).
- Renaming the skill or restructuring the trigger phrase table.
- Changing the cross-platform-docs convention (that lives in the skill body and is fine).
- Building `install.ps1` or any cross-platform install work — that's `[install-ps1]`, downstream of this task.

## Eval set composition (20 queries)

**10 should-trigger queries**, distributed:

| Bucket | Count | Examples (sketch — final list lives in `.tasks/active-platform-eval/eval-set.json`) |
|---|---|---|
| Windows-explicit | ≥3 | "мы на винде, дай команду установки X", "I'm on Windows, give me the README quick-start", "сейчас под виндой, как поставить Y через scoop" |
| Linux-explicit | ≥3 | "we're on a Linux box, write a one-liner для systemd unit", "I'm on Ubuntu prod, как настроить cron", "переключись на линукс, нужен install для apt" |
| macOS-explicit | ≥3 | "я на маке, как поставить через brew", "I'm on a Mac, how do I configure zsh prompt", "switch to macOS, дай команду для plist" |
| Implicit shell context | rest | "give me README quick-start for a CLI tool", "напиши one-liner установки", "how do I run this in a fresh terminal" — should still trigger because output is shell-bound, even without OS-trigger word |

**10 should-not-trigger queries** — near-misses, not obvious negatives:

| Bucket | Count | Sketch |
|---|---|---|
| Library API questions | 3 | "как использовать `subprocess.run` в Python", "what does `os.path.join` do" — touches shell-adjacent territory but is code-question, not produce-commands-for-user |
| Architecture / design talk | 2 | "should I use Redis or Postgres for X", "explain microservices vs monolith" — no command output |
| README content questions (non-commands) | 2 | "what should the description section of a README contain", "how do I write a good API doc" — README *meta*, not quick-start |
| Per-question override tests | 2 | "how would this look on the prod box, which is Ubuntu" — should trigger (per-question), but **must not flip the session-wide default away from current**. The skill body addresses this; eval should test the description doesn't over-trigger and switch sessions on stray "on Ubuntu" mentions. *Mark as should-not-trigger session-switch; ambiguous for skill activation — TBD during eval set finalization.* |
| Genuinely unrelated baseline | 1 | "explain the halting problem" — sanity check that the negative bucket isn't all subtle-near-misses |

Final eval set committed at `.tasks/active-platform-eval/eval-set.json` before the run starts (so the eval is reproducible from git).

## Workflow (parallel)

```
┌─ MAIN AGENT ─────────────────────────────────┐
│                                              │
│  1. Snapshot active-platform skill           │
│  2. Build + commit eval set                  │
│  3. Launch run_loop.py in background ────────┼─→ ┌─ BACKGROUND ──┐
│                                              │   │ run_loop.py   │
│  4. (parallel) manual body sweep:            │   │ ~5-10 min     │
│     — re-read SKILL.md fresh                 │   │ 5 iter × 20q  │
│     — note WSL gap, BSD-coreutils gap,       │   │ × 3 reps,     │
│       ambiguity policy clarity               │   │ train/test    │
│     — draft revised body sections            │   │ split         │
│                                              │   └───────────────┘
│  5. ← run_loop completes → read              │            │
│     best_description from output JSON        │ ←──────────┘
│                                              │
│  6. Apply best_description to frontmatter   │
│  7. Apply body sweep edits                   │
│  8. Bump version 1.0.0 → 1.1.0               │
│  9. Rebuild dist/, reinstall                 │
│ 10. Write .wiki/concepts/active-platform-    │
│     eval.md (results + before/after)         │
│ 11. Update wiki index + log                  │
│ 12. Update STATUS.md (both ⚪ → 🟢)          │
│ 13. Commit, ask before push                  │
│                                              │
└──────────────────────────────────────────────┘
```

The two halves are independent: `run_loop.py` only touches the description field via its own iteration loop; manual body edits don't conflict because they target the body. Both converge in step 6-7 where I apply the diffs in sequence.

## Tooling concerns

**`run_loop.py` location:** `<skill-creator-cache>/skills/skill-creator/scripts/run_loop.py`. Need to invoke `python -m scripts.run_loop` from that directory.

**Model ID for the loop:** `claude-opus-4-7` (current session model — the loop's triggering test should match the model the user actually experiences).

**`claude` CLI dependency:** `run_loop.py` shells out to `claude -p` to test triggering. Need to confirm the `claude` CLI is on PATH before kickoff. If missing, fallback to manual single-pass description rewrite.

**Workspace path:** `.tasks/active-platform-eval/` — gitignore the per-iteration subdirectories (transient outputs, benchmarks, HTML reports), commit only `eval-set.json` and the final summary in `.wiki/concepts/active-platform-eval.md`. Add a `.gitignore` line scoped to that workspace.

## Versioning + manifest

`version: 1.0.0 → 1.1.0` — MINOR per project-discipline Rule 3:
- Description rewrite = improved triggering, behavior compatible.
- Body additions (WSL clarity, BSD/macOS expansion, ambiguity policy refinement) = added capability, no removed coverage.

`bootstrap-manifest.md` (in projects bootstrapped from this repo) lists `active-platform: 1.0.0` — bumping doesn't propagate retroactively, but new bootstraps after this commit will pick up `1.1.0` automatically via the manifest-write step.

## Spec self-review

- ☑ No "TBD" — except one explicit ambiguity flag in the eval set (per-question override category) that's resolved during eval set finalization, not deferred.
- ☑ Internal consistency: Workflow step 6 applies `best_description`, step 7 applies body edits, step 8 bumps version — all in sequence, no contradiction.
- ☑ Scope: focused on `active-platform`. `[install-ps1]` mentioned only as downstream sequencing; explicitly out of scope.
- ☑ Ambiguity: `should-not-trigger` for "per-question override" cases is the one tricky spot. Resolution: those queries should trigger the skill (the body explicitly handles per-question), but should NOT flip the session-wide default. Eval can only measure trigger-or-not, not session-state mutation — so mark them `should_trigger: true` and rely on the body content to handle the no-flip behavior. Documented in eval set comments.

## Connection downstream

After this task:
- `[install-ps1]` becomes the obvious next item (cross-platform install scripts pair with the now-better-triggered active-platform skill).
- `active-platform-tuning-v2` may emerge later if real-world misfires surface gaps the synthetic eval missed; tracked as a fresh task at that time, not now.

## See also

- [active-platform-decision](active-platform-decision.md) — original decision to make `active-platform` a skill (not memory); open questions list (WSL, macOS coverage) is what this eval surfaces and resolves
- [skill-versioning](skill-versioning.md) — semver discipline that drives the 1.0.0 → 1.1.0 bump
- [bootstrap-skill-deps-check](bootstrap-skill-deps-check.md) — `active-platform` is one of the 9 canonical fulfillers in the bootstrap dependency map; description tuning here doesn't change its `kind: skill` classification