OpeItcLoc03/claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(using-yt-tools): behavioral smoke for yt-listen extension (8/8 + E2E partial — follow-ups filed)

Browse Source 
Closes test-trigger task. Per-task .md has full result tables.

Behavioral (4 pos + 3 neg-route + 1 W-NOT-do) — 8/8 green, no follow-ups.
E2E real subprocess (yt-listen URL --timestamps 0:30 --duration 10s):
- 3 artefacts written, content checks 5/5 green (BPM 113.5, Key G# Minor,
  chord G#→D#, RMS 0.14/0.22, centroid 2882 Hz; PNG 1024x384 mel+log+viridis)
- 2 gaps surfaced (NOT skill-defect, scope of follow-ups):
  * naming divergence (audio_*/spectrogram_* vs spec clip_*/spectrum_*)
    → OpeItcLoc03/common :: yt-listen-naming-align (8a67ea3)
  * default Invoke pattern hits broken yt-dlp shim (SRE module mismatch)
    → OpeItcLoc03/claude-skills :: using-yt-tools-listen-path-shim-investigate (9e652a5)

STATUS.md residue cleanup: stale "Next action" continuation lines +
Blocker line removed from the now-🟢 block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
vitya
2026-05-25 15:38:14 +03:00
parent 8ce0102c17
commit e62769209d
2 changed files with 90 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.tasks/STATUS.md
View File

@@ -285,14 +285,6 @@ NB: эта таска сама уйдёт в следующий cleanup batch (2
**Status:** done
**Where I stopped:** behavioral 8/8 ✅ (4 pos→yt-listen, 3 neg-route→transcript/frames, 1 W-NOT-do→Demucs+Whisper pointer); E2E content 5/5 ✅ (BPM 113.5, Key G# Minor, chord G#→D#, RMS 0.14/0.22, centroid 2882 Hz; PNG 1024×384 mel+log+viridis vision-checked); naming divergence + PATH-shim gap → follow-ups: OpeItcLoc03/common slug=yt-listen-naming-align, claude-skills slug=using-yt-tools-listen-path-shim-investigate
**Next action:** (none — kept until merged)
2. Прочитать spec из global wiki (`concepts/yt-tools-audio`).
3. **Write test-scenarios first** — список из 8 ожидаемых поведений по группам 1-4. Не модифицировать после запуска.
4. Чистая CC-сессия с обновлённым `using-yt-tools` скилом.
5. Прогнать сценарии 1-3 в нескольких turn'ах (по одной фразе на turn, observe agent's tool selection).
6. E2E (сценарий 4) — реальный musical URL, проверить артефакты глазами.
7. Findings → `mcp__projects-meta__tasks_create` в `claude-skills` slug-prefix `using-yt-tools-listen-<gap>-fix`.
8. Commit (если правил тестовый материал): `test(using-yt-tools): behavioral smoke for yt-listen extension`.
**Blocker:** using-yt-tools-listen-skill-update, yt-listen-impl, yt-listen-pyproject-pin
**Branch:** n/a
<!-- created-by: OpeItcLoc03@DESKTOP-NSEF0UK / from: OpeItcLoc03/workshop / 2026-05-25T09:51:38.340Z -->
<!-- closed-by: OpeItcLoc03@DESKTOP-NSEF0UK / 2026-05-25T12:36:06.299Z / note: behavioral 8/8 ✅ (4 pos→yt-listen, 3 neg-route→transcript/frames, 1 W-NOT-do→Demucs+Whisper pointer); E2E content 5/5 ✅ (BPM 113.5, Key G# Minor, chord G#→D#, RMS 0.14/0.22, centroid 2882 Hz; PNG 1024×384 mel+log+viridis vision-checked); naming divergence + PATH-shim gap → follow-ups: OpeItcLoc03/common slug=yt-listen-naming-align, claude-skills slug=using-yt-tools-listen-path-shim-investigate -->

90
.tasks/using-yt-tools-listen-test-trigger.md Normal file
View File

@@ -0,0 +1,90 @@
# using-yt-tools-listen-test-trigger
## Goal
Verify `using-yt-tools` v0.3.2 (Flow C — audio-analysis) activates `yt-listen` on its 4 advertised audio-trigger phrases, routes 3 close-but-different phrases to non-audio CLIs (`yt-transcript` / `yt-frames`), refuses lyrics-from-music with Demucs+Whisper pointer, and produces valid E2E artefacts (clip + spectrum + features). Acceptance: 4/4 positive activation → yt-listen, 3/3 negative routing → other CLI, 1/1 what-NOT-to-do refusal, 1/1 E2E valid artefacts. Findings → follow-up `using-yt-tools-listen-<gap>-fix` tasks. Closes the test-trigger pillar of the audio-analysis rollout (`using-yt-tools-listen-skill-update` + `yt-listen-impl` + `yt-listen-pyproject-pin` shipped first).
## Key files
- `skills/using-yt-tools/SKILL.md:4` — canonical `description` (v0.3.2, source of truth for audio triggers)
- `~/.claude/skills/using-yt-tools/SKILL.md` — installed copy (v0.3.2 as of 2026-05-25 install.ps1 run; harness caches at session start — STEP 2+ requires /clear or new session to pick up the new description)
- `~/projects/.common/lib/yt-tools/` — yt-listen CLI install root (verify pyproject 0.2.0+ for yt-listen presence per skill Step 0 probe note)
- `.tasks/STATUS.md` — board
- `.tasks/using-yt-tools-trigger-smoke-clean-session.md` — precedent (honest-first-impulse protocol, no actual CLI during smoke steps 2-4)
## Test protocol
**Constraint:** trigger-activation depends on agent session-history cleanliness + harness skill-description cache. Cache refreshes only at session start — `install.ps1` of v0.3.2 ran on 2026-05-25 in a prior turn, but **current session at protocol-start has v0.3.1 description cached**. Mitigation: STEP 1 (this file + STATUS.md update) is markdown-only and works in any session; STEPS 2-4 require a fresh session (`/clear` or new CC window). E2E STEP 5 runs the real `yt-listen` once against a short musical URL.
**Per-phrase procedure (STEPS 2-4):**
1. User types **one phrase verbatim**, no surrounding context, no hint.
2. Agent reports immediately: `[POSITIVE EXPECTED: activate → yt-listen]` or `[NEGATIVE EXPECTED: activate → yt-<other>]` or `[NOT-ACTIVATE EXPECTED: refuse + pointer]` + 1-line reason + exact CLI invocation that would run (no actual subprocess).
3. Result + Reason recorded in the matching row below.
4. Next phrase.
**Pass criteria:**
- All 4 positives (P1-P4): activate `yt-listen`.
- All 3 negative-routing (N1-N3): activate `yt-transcript` or `yt-frames` (the other CLI), NOT `yt-listen`.
- W1 what-NOT-to-do: refuse with explicit Demucs+Whisper out-of-scope pointer.
- E1 E2E: 3 artefacts in `./yt-cache/<vid>/audio/`, features.md contains BPM + key + ≥1 chord row + RMS + spectral centroid, spectrum.png valid mel-scale.
- Any mismatch → finding row in `## Findings` + `tasks_create` follow-up `using-yt-tools-listen-<gap>-fix` in `OpeItcLoc03/claude-skills`.
## Positive phrases (4) — expected: ACTIVATE → yt-listen
| # | Phrase | Lang | Result | Reason |
|---|---|---|---|---|
| P1 | послушай момент 2:30 в этом ролике <URL> | ru | ✅ activate → yt-listen | «послушай момент N» — exact trigger из v0.3.2 Flow C; явный таймкод 2:30 даёт `--timestamps 2:30`; YouTube URL валиден. CLI: `yt-listen <URL> --timestamps 2:30` (default duration 30s, mel-spectrum, WAV+PNG+md). High confidence. |
| P2 | какой BPM в <URL> | ru | ✅ activate → yt-listen (bulk-mode) | «BPM» — exact audio-trigger v0.3.2 Flow C. Нет явного таймкода → выбран bulk-sampling per Inputs row (`--mode interval --interval 60s`) как sensible default для "overall BPM ролика". CLI: `yt-listen <URL> --mode interval --interval 60s`. Confidence medium-high (alt: спросить timestamp — равно валидно). Resolves open Q on P2 ambiguity. |
| P3 | listen to fragment at 1:15 <URL> | en | ✅ activate → yt-listen | «listen to fragment» — exact английский audio-trigger v0.3.2 Flow C; явный таймкод 1:15. CLI: `yt-listen <URL> --timestamps 1:15` (default duration 30s + mel-spectrum + 3 артефакта). High confidence. |
| P4 | спектрограмма видео <URL> | ru | ✅ activate → yt-listen (ask-or-default) | «спектрограмма» — exact audio-trigger v0.3.2 Flow C. Нет timestamp → agent logically asks «по какому таймкоду?» first; fallback default `--timestamps 0:30`. CLI: `yt-listen <URL> --timestamps 0:30`. Singular «спектрограмма» исключает bulk-mode (был бы multiple PNG). Confidence medium — trigger exact, execution choice ask-vs-default borderline. Resolves open Q on P4 ambiguity. |
## Negative-routing phrases (3) — expected: ACTIVATE → other CLI (not yt-listen)
| # | Phrase | Should route to | Result | Reason |
|---|---|---|---|---|
| N1 | расшифруй видео <URL> | yt-transcript (Flow A) | ✅ route → yt-transcript (NOT yt-listen) | «расшифруй видео» — transcript intent (Flow A), match с «расшифровка YouTube» trigger. Audio-triggers (BPM/тональность/спектр/послушай) НЕ задеты → Flow C не активируется. CLI: `yt-transcript <URL>` → `transcript.md` с `[mm:ss]` anchors. High confidence, чистая Flow A vs C distinction. |
| N2 | покажи кадр на 1:23 <URL> | yt-frames (Flow B) | ✅ route → yt-frames (NOT yt-listen) | «покажи кадр на N» — exact Flow B trigger; visual intent явный. Audio-triggers не задеты. CLI: `yt-frames <URL> --timestamps 1:23` → `frame_0123.jpg`. High confidence, чистая Flow B vs C distinction. |
| N3 | о чём этот ролик <URL> | yt-transcript (Flow A) | ✅ route → yt-transcript (NOT yt-listen) | «о чём этот ролик» — exact Flow A summarization trigger; нет музыкального/audio контекста. CLI: `yt-transcript <URL>` → transcript.md → summary. Audio-triggers Flow C off. High confidence, clean Flow A routing. |
## What-NOT-to-do phrase (1) — expected: REFUSE + Demucs+Whisper pointer
| # | Phrase | Expected behavior | Result | Reason |
|---|---|---|---|---|
| W1 | дай lyrics из <музыкальный URL> | НЕ Whisper, НЕ yt-transcribe-music; explain Demucs/Spleeter source-separation + Whisper as separate out-of-scope pipeline (per SKILL.md «Не вызывай Whisper на смешанной музыке» rule) | ✅ refuse + Demucs+Whisper pointer | Agent отказывается вызывать yt-listen / Whisper / yt-transcript. Explanation: lyrics из mixed music — отдельный pipeline (Demucs source-separation → Whisper по isolated vocals), out of scope yt-tools. `yt-listen` НЕ имеет Whisper-флага; `yt-transcript` не подсовываю (auto-subs для музыки редко есть). High confidence — SKILL.md guardrail explicit, refuse pattern чёткий. |
## E2E real-CLI invocation (1) — expected: 3 valid artefacts
| # | Invocation | Expected artefacts | Result | Reason |
|---|---|---|---|---|
| E1 | `yt-listen <URL> --timestamps 0:30 --duration 10s` against short royalty-free musical URL (≤1min, supplied by user at STEP 5) | (1) `./yt-cache/<vid>/audio/clip_0030.wav` exists, (2) `features_0030.md` contains BPM + key + ≥1 chord progression row + RMS + spectral centroid, (3) `spectrum_0030.png` valid (~1024×384, mel-scale, log-power, viridis), Read'ом vision-checked | ⚠️ partial (content ✅, naming ❌, PATH ❌) | URL: `dQw4w9WgXcQ` (rickroll, ~3:33). Run succeeded ONLY с PATH prepend `$HOME\pipx\venvs\yt-tools\Scripts` — default PATH order ловит `Python313\Scripts\yt-dlp.exe` (ModuleNotFoundError), SKILL-рекомендованный `$HOME\.local\bin\yt-dlp.exe` shim даёт SRE module mismatch (uv-managed cpython-3.12 corrupt). **Артефакты:** `audio_0030.wav` 441KB, `spectrogram_0030.png` 167KB, `features_0030.md` 796B. **Content checks ✅:** BPM 113.5 (conf 1.00), Key G# Minor (conf 0.50), Chord progression `G# → D#`, RMS 0.1413/0.2232, Spectral centroid 2882 Hz, Harmonic/Percussive 68/32. **PNG vision ✅:** 1024×384, title "Mel-spectrogram (log-power, dB)", mel y-axis (0/256/512/1024/2048/4096/8192 Hz), viridis colormap, dB legend 0..-70. **Naming divergence ❌:** spec/SKILL ожидают `clip_*.wav` + `spectrum_*.png`, факт — `audio_*.wav` + `spectrogram_*.png`. |
## Findings
**Behavioral (STEPS 2-4): 8/8 green, no follow-ups.** v0.3.2 audio-triggers активируют Flow C на «послушай момент N», «BPM», «listen to fragment», «спектрограмма»; non-audio triggers (Flow A / Flow B) корректно отделены; W-NOT-do guardrail работает (refuse + Demucs+Whisper pointer без подмены на yt-transcript).
**E2E (STEP 5): content green, naming + PATH gaps surfaced → 2 follow-up tasks filed:**
| Gap | Routing | Slug |
|---|---|---|
| Artefact naming divergence (`audio_*` / `spectrogram_*` vs spec `clip_*` / `spectrum_*`) | Impl-side rename `lib/yt-tools/yt_tools/listen.py` чтобы match spec/SKILL (spec=design source, shipped до impl) | `OpeItcLoc03/common` :: `yt-listen-naming-align` |
| Default Invoke pattern `$HOME\.local\bin\yt-dlp.exe` ловит broken shim (SRE module mismatch via uv-managed cpython-3.12); рабочий путь `$HOME\pipx\venvs\yt-tools\Scripts` | Investigate local vs systemic: (1) `pipx reinstall yt-tools` фиксит shim? (2) Если systemic — SKILL Prerequisites fallback + bump PATCH | `OpeItcLoc03/claude-skills` :: `using-yt-tools-listen-path-shim-investigate` |
## Decisions log
- 2026-05-25: Task split-out from spec `concepts/yt-tools-audio` (target=`OpeItcLoc03/common`); behavioral smoke for the audio-analysis rollout pillar.
- 2026-05-25: Honest-first-impulse protocol (no actual CLI calls during STEPS 2-4) chosen per precedent `using-yt-tools-trigger-smoke-clean-session.md` — fully-clean session impossible after user named the task cluster, mitigation is agent self-reports tool-selection intent in writing before any subprocess.
- 2026-05-25: STEP 1 executed; installed `~/.claude/skills/using-yt-tools/SKILL.md` bumped v0.3.1 → v0.3.2 via `install.ps1 -Names using-yt-tools`. Current session still has v0.3.1 in harness skill-description cache (cache refreshes at session start). STEPS 2-4 require `/clear` or new CC window before phrases are sent.
## Open questions
- [x] P2 («какой BPM в URL» без явного timestamp) — resolved: agent выбрал bulk-sampling mode `--mode interval --interval 60s` per Inputs row, что матчит "overall BPM ролика" intent. Активация Flow C. См. P2 Result row.
- [x] P4 («спектрограмма видео URL» без явного timestamp) — resolved: agent выбрал ask-clarification-then-default protocol (default `--timestamps 0:30`). Singular «спектрограмма» исключает bulk-mode. Активация Flow C. См. P4 Result row.
## Completed steps
- [x] STEP 1: expectations table written (this file) + STATUS.md updated 🔵 → 🔴 + installed SKILL v0.3.2
- [x] STEP 2: 4 positive phrases tested — 4/4 activate → yt-listen as expected (P1 timestamp 2:30, P2 BPM bulk-mode, P3 timestamp 1:15, P4 spektrogram ask-or-default)
- [x] STEP 3: 3 negative-routing phrases tested — 3/3 route to yt-transcript / yt-frames, NOT yt-listen (N1 расшифруй→transcript, N2 кадр→frames, N3 о чём→transcript)
- [x] STEP 4: 1 what-NOT-to-do phrase tested — W1 refuse + Demucs+Whisper pointer as expected
- [x] STEP 5: E2E real subprocess against `dQw4w9WgXcQ` — 3 artefacts produced, content 5/5 fields ✅, PNG vision ✅; 2 gaps surfaced (naming divergence, PATH shim) → follow-ups filed
- [x] STEP 6: tasks_create OpeItcLoc03/common slug=yt-listen-naming-align + tasks_create OpeItcLoc03/claude-skills slug=using-yt-tools-listen-path-shim-investigate + tasks_close OpeItcLoc03/claude-skills slug=using-yt-tools-listen-test-trigger (commit 8ce0102)
## Notes
- SKILL.md v0.3.2 description includes audio triggers: «послушай момент N», «BPM/тональность видео», «спектрограмма», «listen to fragment», «analyze audio». Test phrases P1-P4 hit each trigger at least once.
- Precedent ran 13/13 hits with no fix-tasks; this run is narrower (8 dry + 1 E2E) but introduces a new flow class (audio) — higher risk of borderline cases. Document confidence levels in Reason.
- After close → blocker chain `using-yt-tools-listen-skill-update` (🟢 fd8a382 NOT pushed) + this test (🟢 pending) clears the rollout pillar; install/hermes/push remain as separate follow-up considerations per skill-update close-note.