claude-skills/.tasks/active-platform-eval.md

active-platform-eval

Goal

Tune active-platform (description + body) using skill-creator's run_loop.py for description optimization in parallel with a manual body sweep. Replaces the placeholder "5 real signals" approach with a 20-query synthetic eval set balanced across Windows / Linux / macOS. Absorbs the two ⚪ tasks [active-platform-tuning] + [active-platform-eval] into one workstream — they were inseparable in practice (eval is the tuning mechanism). Bumps version: 1.0.0 → 1.1.0 (MINOR — improved triggering + expanded body coverage; behavior compatible).

Key files

• skills/active-platform/SKILL.md — frontmatter description (run_loop output) + body (manual sweep) + version bump
• dist/active-platform.skill — rebuild after edits
• ~/.claude/skills/active-platform/ — reinstall
• .tasks/active-platform-eval/eval-set.json — 20-query trigger eval set, committed before kickoff
• .tasks/active-platform-eval/iteration-N/ — workspace for run_loop iterations (gitignored; transient)
• .tasks/active-platform-eval/.gitignore — scopes the iteration subdirs out of git
• .wiki/concepts/active-platform-eval-design.md — design doc (this task's spec; already written)
• .wiki/concepts/active-platform-eval.md — final report (eval set summary, before/after scores, body diff summary)
• .wiki/concepts/active-platform-decision.md — open-questions list (WSL, macOS coverage) is exactly what this eval resolves; cross-link
• .wiki/index.md — add new concept entry
• .wiki/log.md — append decision line
• .tasks/STATUS.md — collapse [active-platform-tuning] + [active-platform-eval] ⚪ blocks into one 🔴 active block, then 🟢 done

Decisions log

• 2026-05-05: combined the two backlog tasks into one. The split was an artefact of the original "wait for production misfires" model; synthetic eval covers the same ground in one pass with controlled cross-platform balance.
• 2026-05-05: chose autoloop (run_loop.py × 5 iterations) in parallel with manual body sweep, per user explicit request ("давай одновременно"). The two halves don't conflict — autoloop only mutates frontmatter description; body sweep only mutates body.
• 2026-05-05: model for run_loop = claude-opus-4-7 (current session model — triggering test must match what the user actually experiences in production).
• 2026-05-05: workspace at .tasks/active-platform-eval/ (project-internal, not ~/AppData/Local/Temp/...) so the eval set lives in git and the workflow is reproducible from any clone. Iteration subdirs gitignored — transient HTML reports + per-iteration outputs would balloon the repo.
• 2026-05-05: eval set composition forced cross-platform balance: ≥3 should-trigger queries per OS (Win/Lin/Mac), rest implicit-shell-context. ≥10 should-not-trigger queries are deliberately near-misses (library API questions, README meta-questions, architecture talk) — only 1 genuinely-unrelated negative as a sanity baseline. Per-question-override cases marked should_trigger: true (the skill should fire, but body handles no-session-flip behavior, which the eval can't measure directly).
• 2026-05-05: spec lives at .wiki/concepts/active-platform-eval-design.md per project-discipline Rule 1 (overrides skill-creator's default docs/superpowers/specs/). Final report goes to a sibling active-platform-eval.md (separate "what we're going to do" vs "what we did").
• 2026-05-05: version bump 1.0.0 → 1.1.0 (MINOR) — added capability (better triggering, broader body coverage), no removed coverage, behavior compatible. Recorded in commit message per Rule 3.
• 2026-05-05: install-ps1 task scope expanded in the same session (paired install.sh + install.ps1, cross-platform parity, --prune flag) — out of scope here, but logged in STATUS.md so we don't lose the requirement when this task closes.

Open questions

• Does the claude CLI exist on PATH? run_loop.py shells out to claude -p for triggering tests. If not on PATH → fall back to manual single-pass description rewrite (still useful, but loses the train/test optimization).
• How does run_loop.py behave on Windows? skill-creator's docs assume bash-like shell for the invocation; may need to invoke through git-bash or adjust working-directory handling.
• Per-question override eval cases — confirmed should_trigger: true (skill fires) but the no-session-flip behavior is body-driven, not measurable by trigger eval. Document in .wiki/concepts/active-platform-eval.md final report so future readers know the eval didn't directly measure that.

Completed steps

• 2026-05-05: read current active-platform/SKILL.md (lines 1-86) and active-platform-decision.md (lines 1-46) to baseline scope and known weak spots (WSL, macOS coverage, trigger phrase comprehensiveness).
• 2026-05-05: invoked superpowers:brainstorming skill, walked through the design choices (combine vs split, autoloop vs manual, parallel vs sequential), reached approval on combined-and-parallel plan.
• 2026-05-05: wrote design doc .wiki/concepts/active-platform-eval-design.md (per project-discipline Rule 1).
• 2026-05-05: pre-flight verified — claude CLI on PATH (C:\nvm4w\nodejs\claude.ps1, Claude Code 2.1.128); run_loop.py present at ~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/skills/skill-creator/scripts/run_loop.py. Both autoloop dependencies satisfied; no fallback to manual single-pass needed.
• 2026-05-05: STATUS.md collapsed the two original ⚪ tasks ([active-platform-tuning] + [active-platform-eval]) into one block, set 🔴 active during design phase, then 🟡 paused at user request before eval-set authorship.
• 2026-05-05: paused at user request — task complete through design + tooling pre-flight; resume point is Q2 (write eval set solo vs HTML-review).
• (further steps appended during implementation when resumed)

Notes

• Skill-creator path on this machine: ~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/skills/skill-creator/. run_loop.py lives under that at scripts/run_loop.py.
• Snapshot the current skill before kickoff: cp -r skills/active-platform .tasks/active-platform-eval/skill-snapshot/ so the run_loop's "old description" baseline matches what we're tuning from, regardless of any edits I make to the body during the parallel sweep.
• run_loop accepts --max-iterations N. Plan = 5 (skill-creator default + sufficient signal for a single description field).
• run_loop output: best_description selected by held-out test score, not train score (skill-creator's anti-overfit choice). Trust the test-score winner unless qualitative review of the score table reveals an obvious flaw.
• After this task: [install-ps1] (already updated in STATUS.md to require paired install.sh + install.ps1 + --prune), then revisit any further skill backlog items.