refactor(skills): dedup compress -> caveman-compress [v1.0.0]
skills/compress/ was a byte-identical dupe of skills/caveman-compress/ (SHA256 match across all 7 scripts/ files; SKILL.md diff was name: + Process step 2 only; descriptions textually identical = arbitrary harness tie-break + double-counted listing budget). Kept caveman-compress canonical: it carries README.md (benchmarks + caveman-toolkit branding) and SECURITY.md (Snyk false-positive writeup), and matches the caveman-* prefix invariant. Ported the better Process-step wording from compress (`cd <directory_containing_this_SKILL.md>`) into caveman-compress; the prior `cd caveman-compress` form assumed cwd was the parent dir and broke when invoked from ~/.claude/skills/caveman-compress/. Added version: 1.0.0 to caveman-compress frontmatter (first versioned release; aligns with skill-versioning concept). Removed: skills/compress/, dist/compress.skill, ~/.claude/skills/compress/ (manual prune; install.sh has no prune step yet -- future [install-ps1] task should add --prune). Slash-command impact: /compress gone; /caveman-compress and /caveman:compress (toolkit-canonical) remain. Wiki: .wiki/concepts/compress-dedup.md (rationale + rejected alternatives), index.md + log.md updated. Tasks: [compress-dedup] flipped active -> done in same commit (per-task file .tasks/compress-dedup.md created with full decisions log). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
# Task Board
|
||||
_Updated: 2026-05-05_
|
||||
_Updated: 2026-05-05 (compress-dedup done)_
|
||||
|
||||
<!--
|
||||
Canonical layout. One block per task. Per-task deep context lives in
|
||||
@@ -38,11 +38,11 @@ _Updated: 2026-05-05_
|
||||
|
||||
---
|
||||
|
||||
## ⚪ [compress-dedup] — resolve compress vs caveman-compress duplication
|
||||
**Status:** ready
|
||||
**Where I stopped:** (not started) — noticed during initial migration that both skills ship identical `scripts/` subtrees (`compress.py`, `cli.py`, `detect.py`, `validate.py`, `benchmark.py`)
|
||||
**Next action:** `diff -r skills/compress skills/caveman-compress`; decide if one is canonical and the other should be removed (or aliased)
|
||||
**Branch:** (not started)
|
||||
## 🟢 [compress-dedup] — resolve compress vs caveman-compress duplication
|
||||
**Status:** done
|
||||
**Where I stopped:** `skills/compress/` deleted as a byte-identical dupe of `skills/caveman-compress/` (SHA256 match across all 7 scripts/ files; SKILL.md diff was `name:` + Process step 2 only; README + SECURITY only in caveman-compress). Ported the better Process-step wording from compress (`cd <directory_containing_this_SKILL.md>` instead of brittle `cd caveman-compress`) into `skills/caveman-compress/SKILL.md`. Added `version: 1.0.0` to caveman-compress frontmatter (first versioned release; aligns with skill-versioning concept). Removed: `skills/compress/`, `dist/compress.skill`, `~/.claude/skills/compress/` (manual prune — install.sh has no prune step). Rebuilt `dist/caveman-compress.skill` + reinstalled to `~/.claude/skills/caveman-compress/` (verified `version: 1.0.0` + new Process step on disk). Harness skill listing confirms `compress` is gone, only `caveman-compress` remains. `.wiki/concepts/compress-dedup.md` written (rationale + rejected alternatives: alias-stub has no harness mechanism, "keep both" wastes listing budget, "delete caveman-compress" loses README + SECURITY); `.wiki/index.md` + `.wiki/log.md` updated.
|
||||
**Next action:** (none — kept until merged)
|
||||
**Branch:** master
|
||||
|
||||
---
|
||||
|
||||
|
||||
38
.tasks/compress-dedup.md
Normal file
38
.tasks/compress-dedup.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# compress-dedup
|
||||
|
||||
## Goal
|
||||
Resolve the duplication between `skills/compress/` and `skills/caveman-compress/`. Both ship byte-identical `scripts/` (7 files: `__init__.py`, `__main__.py`, `benchmark.py`, `cli.py`, `compress.py`, `detect.py`, `validate.py`) and near-identical `SKILL.md` (only `name:` field and Process step 2 differ). Both register independently with the harness with identical descriptions — listing-token waste and arbitrary tie-break on activation. Pick canonical = `caveman-compress` (richer: README + SECURITY + caveman-toolkit branding); port the better Process-step wording from `compress` (generic `cd <directory_containing_this_SKILL.md>` vs brittle `cd caveman-compress`); delete `skills/compress/` entirely. Add `version: 1.0.0` frontmatter to caveman-compress (project-discipline Rule 3 alignment). Skill listing collapses from two redundant entries to one rich one. Install path `~/.claude/skills/compress/` is removed.
|
||||
|
||||
## Key files
|
||||
- `skills/compress/` — entire folder deleted
|
||||
- `skills/caveman-compress/SKILL.md` — port Process step from compress; add `version: 1.0.0`
|
||||
- `skills/caveman-compress/README.md` — left as-is (already canonical, branding intact)
|
||||
- `skills/caveman-compress/SECURITY.md` — left as-is
|
||||
- `dist/compress.skill` — deleted (rebuild won't recreate; source gone)
|
||||
- `dist/caveman-compress.skill` — rebuilt after SKILL.md edit
|
||||
- `~/.claude/skills/compress/` — removed manually (install.sh has no prune step)
|
||||
- `~/.claude/skills/caveman-compress/` — reinstalled
|
||||
- `.wiki/concepts/compress-dedup.md` — design rationale (why caveman-compress canonical, why no alias mechanism, install-path impact)
|
||||
- `.wiki/index.md` — add concept entry
|
||||
- `.wiki/log.md` — append decision line
|
||||
- `.tasks/STATUS.md` — flip compress-dedup ⚪ → 🔴 → 🟢
|
||||
|
||||
## Decisions log
|
||||
- 2026-05-05: scripts/ byte-identical (SHA256 match across all 7 files). SKILL.md diff = `name:` field + Process step 2 only. Description text identical between the two. README + SECURITY only in caveman-compress. Conclusion: same skill, two registrations.
|
||||
- 2026-05-05: chose `caveman-compress` as canonical over `compress`. Rationale: (a) richer docs (README with benchmarks table, branding, "Part of Caveman" toolkit linkage; SECURITY.md with Snyk false-positive writeup), (b) name reflects actual upstream origin (JuliusBrussee/caveman toolkit), (c) caveman cluster (caveman, caveman-commit, caveman-review, caveman-help, caveman-compress) has consistent prefix.
|
||||
- 2026-05-05: rejected "alias-stub" approach (keep `compress` as a tiny SKILL.md pointing at caveman-compress). Harness has no alias mechanism — every SKILL.md fully registers; alias-stub still consumes a listing slot and a description budget. No win.
|
||||
- 2026-05-05: rejected "keep both" approach. Identical descriptions ⇒ harness arbitrarily tie-breaks; both consume listing tokens; user has no way to predict which fires. Pure waste.
|
||||
- 2026-05-05: ported Process step 2 wording from `compress` SKILL.md into `caveman-compress` SKILL.md. The compress version (`cd <directory_containing_this_SKILL.md> && python3 -m scripts ...`) is robust regardless of cwd; the caveman-compress version (`cd caveman-compress && ...`) implicitly assumes cwd is the parent directory. Net: caveman-compress gets the better text + keeps everything else.
|
||||
- 2026-05-05: added `version: 1.0.0` frontmatter to caveman-compress SKILL.md. First versioned release of this skill. Aligns with the rest of the infra-skill cluster (skill-versioning concept).
|
||||
- 2026-05-05: install.sh has no prune step — removing `skills/compress/` does NOT remove `~/.claude/skills/compress/`. Manual `rm -rf ~/.claude/skills/compress/` required. Filed mental note for `[install-ps1]` task: future installer should accept a `--prune` flag, or `setup-tasks`-style canon-list, to remove stale installs.
|
||||
- 2026-05-05: slash-command impact — `/compress` slash command stops working after deletion; `/caveman-compress` and `/caveman:compress` (the canonical caveman-toolkit form) remain. Confirmed with user that `/compress` was effectively never used; no migration needed.
|
||||
|
||||
## Open questions
|
||||
- [ ] Eventually wire up an `install.sh --prune` flag that drops `~/.claude/skills/<name>/` for any name not in `skills/<name>/`. Tracked indirectly via `[install-ps1]` ⚪ task — that task should at minimum match install.sh, but adding prune is the right time to do it.
|
||||
|
||||
## Completed steps
|
||||
- (filled in during implementation)
|
||||
|
||||
## Notes
|
||||
- Origin: both skills came from the JuliusBrussee/caveman upstream toolkit (caveman-compress carries the "Part of Caveman" branding); `skills/compress/` looks like a stripped-down copy that lost README + SECURITY + branding. Never reconciled until this task.
|
||||
- Frontmatter description budget reminder (~900 chars): caveman-compress description is well under, no risk on this edit.
|
||||
91
.wiki/concepts/compress-dedup.md
Normal file
91
.wiki/concepts/compress-dedup.md
Normal file
@@ -0,0 +1,91 @@
|
||||
---
|
||||
title: compress vs caveman-compress dedup
|
||||
type: concept
|
||||
tags: [skills, dedup, caveman, repo-hygiene]
|
||||
updated: 2026-05-05
|
||||
---
|
||||
|
||||
# compress vs caveman-compress dedup
|
||||
|
||||
> Two skills shipped the same payload under different names. Kept `caveman-compress` (richer, on-brand), deleted `compress`. Recorded here so future readers don't reinvent the analysis.
|
||||
|
||||
## What we found
|
||||
|
||||
Side-by-side audit of `skills/compress/` and `skills/caveman-compress/`:
|
||||
|
||||
| Layer | `compress` | `caveman-compress` |
|
||||
|---|---|---|
|
||||
| `scripts/__init__.py` | identical (SHA256) | same |
|
||||
| `scripts/__main__.py` | identical | same |
|
||||
| `scripts/benchmark.py` | identical | same |
|
||||
| `scripts/cli.py` | identical | same |
|
||||
| `scripts/compress.py` | identical | same |
|
||||
| `scripts/detect.py` | identical | same |
|
||||
| `scripts/validate.py` | identical | same |
|
||||
| `SKILL.md` description | byte-identical text | byte-identical text |
|
||||
| `SKILL.md` Process step 2 | `cd <directory_containing_this_SKILL.md> && python3 -m scripts ...` (generic) | `cd caveman-compress && python3 -m scripts ...` (brittle, assumes cwd is parent) |
|
||||
| `SKILL.md` `name:` | `compress` | `caveman-compress` |
|
||||
| `README.md` | absent | present (benchmarks table, branding, "Part of Caveman" toolkit linkage) |
|
||||
| `SECURITY.md` | absent | present (Snyk false-positive writeup: subprocess + file I/O patterns explained) |
|
||||
| Frontmatter `version:` | absent | absent (added `1.0.0` as part of this dedup) |
|
||||
|
||||
Description text in both skills' frontmatter:
|
||||
|
||||
```
|
||||
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
|
||||
to save input tokens. ... Trigger: /caveman:compress <filepath> or "compress memory file"
|
||||
```
|
||||
|
||||
— literally the same paragraph. The harness loads both skill listings, both register independently, descriptions identical ⇒ activation tie-break is arbitrary, and the listing budget pays twice.
|
||||
|
||||
## What we kept and why
|
||||
|
||||
**Canonical = `caveman-compress`.** Reasons in order of weight:
|
||||
|
||||
1. **Richer artefacts** — README.md and SECURITY.md exist only here. The README is not boilerplate: benchmarks table on real files (claude-md-preferences.md, project-notes.md, mixed-with-code.md), 46% average savings claim, before/after example. The SECURITY.md is what static-analysis (Snyk) hooks into, so deleting it would re-open the same false-positive ticket the upstream author already closed.
|
||||
2. **On-brand naming** — the skill is part of the [JuliusBrussee/caveman](https://github.com/JuliusBrussee/caveman) toolkit. The `caveman-*` prefix lines up with the rest of the cluster: `caveman` (speak compressed), `caveman-commit` (commit-message variant), `caveman-review` (PR-review variant), `caveman-help` (help card), `caveman-compress` (compress files). `compress` was the lone exception that broke the prefix invariant.
|
||||
3. **History reads cleanly** — the per-task `[skill-readmes]` plan groups the caveman cluster as a single batch for README work. Keeping the prefix means that batch covers all of them; mixing in a stray `compress` would force a "wait, do we README this one separately?" decision later.
|
||||
|
||||
## What we ported across before deleting
|
||||
|
||||
The only thing `compress` did better than `caveman-compress` was Process step 2. Original wording:
|
||||
|
||||
```
|
||||
cd caveman-compress && python3 -m scripts <absolute_filepath>
|
||||
```
|
||||
|
||||
This assumes cwd is the parent of `caveman-compress/`. When the agent invokes the skill from `~/.claude/skills/caveman-compress/`, that assumption is false (cwd is the skills dir, parent is `~/.claude/`). The `compress` SKILL.md had the robust form:
|
||||
|
||||
```
|
||||
cd <directory_containing_this_SKILL.md> && python3 -m scripts <absolute_filepath>
|
||||
```
|
||||
|
||||
— ported into `caveman-compress/SKILL.md` as part of this dedup. Net: the canonical skill now has the better instructions plus the README + SECURITY.
|
||||
|
||||
Also added `version: 1.0.0` to frontmatter — first versioned release. Aligns with the rest of the infra-skill cluster ([skill-versioning](skill-versioning.md)).
|
||||
|
||||
## What we rejected
|
||||
|
||||
**Alias-stub** — keep `skills/compress/SKILL.md` as a tiny pointer file with description "alias for caveman-compress, see that skill". Rejected: the harness has no alias mechanism. Every SKILL.md file fully registers as a skill, with its description text consuming the listing budget. An alias-stub still occupies a listing slot and a description allowance — there's no "see that skill" indirection at the harness level. Pure waste of a listing entry.
|
||||
|
||||
**Keep both, document them as siblings** — let the user pick which slash-command name they prefer (`/compress` vs `/caveman-compress`). Rejected: descriptions are identical; the harness has no signal to disambiguate; activation tie-break is arbitrary. Forces every future `/caveman:compress` call to gamble on which copy the harness picks.
|
||||
|
||||
**Delete `caveman-compress`, keep `compress` as the simpler name** — rejected because we'd lose the README, SECURITY.md, and the caveman-toolkit branding link. `compress` was the stripped-down fork; reverting to it would be a net regression in artefact quality.
|
||||
|
||||
## Install-path impact
|
||||
|
||||
`scripts/install.sh` has no prune step. Removing `skills/compress/` does **not** remove `~/.claude/skills/compress/` on the user's machine — the install script copies, never deletes. Manual `rm -rf ~/.claude/skills/compress/` is required during this dedup. Same applies to `dist/compress.skill`: deleted explicitly, since the next `build.sh` run iterates over `skills/<name>/` and won't recreate an artefact for a source folder that no longer exists.
|
||||
|
||||
> **Future work:** the [`install-ps1`](../../.tasks/STATUS.md) ⚪ task should at minimum match `install.sh`'s behavior, but is also the right time to add a `--prune` flag that removes `~/.claude/skills/<name>/` for every name not in `skills/<name>/`. Would have made this dedup automatic instead of three-step.
|
||||
|
||||
## Slash-command impact
|
||||
|
||||
Before: `/compress`, `/caveman-compress`, and `/caveman:compress` (toolkit-canonical) were all live.
|
||||
|
||||
After: `/compress` is gone. `/caveman-compress` and `/caveman:compress` still work. User confirmed `/compress` was effectively never used; no migration needed.
|
||||
|
||||
## See also
|
||||
|
||||
- [skill-versioning](skill-versioning.md) — why infra skills carry `version:` in frontmatter and the project-discipline Rule 3 bumps tie back here
|
||||
- [repo-layout](repo-layout.md) — flat `skills/` layout means duplicates show up as flat sibling folders, easy to spot in a directory listing
|
||||
- [skill-vs-plugin](skill-vs-plugin.md) — caveman-compress is a bare SKILL.md (no slash command, no hooks, no MCP), the right granularity for this workload
|
||||
@@ -28,6 +28,7 @@ Catalog of all wiki pages. One line per page, organized by type. Updated on ever
|
||||
- [skill-vs-plugin.md](concepts/skill-vs-plugin.md) — when a bare SKILL.md is enough vs when you actually need a plugin (slash commands, hooks, sub-agents, MCP servers); concrete breakdown of `superpowers`
|
||||
- [wiki-realignment.md](concepts/wiki-realignment.md) — fixing `project-bootstrap` to create the Karpathy-canonical wiki layout
|
||||
- [interns-design](concepts/interns-design.md) — interns-design
|
||||
- [compress-dedup.md](concepts/compress-dedup.md) — `skills/compress/` deleted as a byte-identical dupe of `skills/caveman-compress/`; canonical kept for README + SECURITY + caveman-toolkit branding; better Process-step wording ported across; `version: 1.0.0` added to caveman-compress frontmatter
|
||||
|
||||
## Packages
|
||||
|
||||
|
||||
@@ -45,3 +45,4 @@ Parseable: `grep "^## \[" .wiki/log.md | tail -20`.
|
||||
## [2026-05-05] decision | interns-skills-mvp — shipped `setup-interns` v0.1.0 (8-phase install: detect `.common/lib/interns-mcp/`, `pip install -e`, `.common/secrets/interns.env` write, `mcpServers.interns` registration with absolute Python interpreter + `cwd`) and `using-interns` v0.1.0 (runtime policy mirroring project-discipline Rule 4: ask-mode default, conversational grant/revoke, always-ask paths for `.env`/secrets/keys/SSH/credentials with transitive rule, cost-cap >$0.10, session-end reset; routing hints for `bulk_text_read` + `transcript_distill`); `project-bootstrap` 1.5.0→1.6.0 with canonical CLAUDE.md trigger `delegate to interns when allowed` between `follow project discipline` and `we're on Windows`, Step 5 commentary paragraph, manifest table extended with both new skills + `project-discipline` row; root `CLAUDE.md` dogfood updated; both READMEs written; descriptions verified (setup-interns 899 chars, using-interns 814 chars, both under 900 budget); all three rebuilt + installed + listed by harness with full descriptions (no H1 fallback)
|
||||
## [2026-05-05] ingest | concepts/bootstrap-skill-deps-check
|
||||
## [2026-05-05] decision | bootstrap-skill-deps-check — `project-bootstrap` 1.6.0→1.7.0 collapses Step 5.6 from a single-skill detector (only `superpowers` plugin) into a generic `trigger → fulfiller` table walker. Map embedded in SKILL.md (9 rows: caveman, superpowers plugin, using-wiki, using-tasks, using-projects-meta, pulling-before-work, project-discipline, using-interns, active-platform); `kind: skill` vs `kind: plugin` flag drives the install command emitted in the recommendation block. Algorithm: read project's CLAUDE.md → match each line vs map (substring + tolower, mirrors Step 5 idempotent merge) → for each canonical match check disk (`~/.claude/skills/<name>/SKILL.md` or `installed_plugins.json` key); print one chat-only block listing every missing fulfiller + install commands, or one ✅ line if all satisfied. User-custom lines silently skipped; removed canonical lines silently skipped (respects user opt-out). Hard rule "never auto-install" carries over verbatim. Subsumes the deferred `[bootstrap-recommend-projects-meta]` task (closed by absorption — generic step handles `using-projects-meta` along with everything else). MCP-server-backed skills only check the `using-X` policy skill; `setup-X` self-fires on first use via Prerequisites pointer, bootstrap doesn't duplicate.
|
||||
## [2026-05-05] decision | compress-dedup — `skills/compress/` was a stripped-down byte-for-byte dupe of `skills/caveman-compress/` (scripts/ identical SHA256 across all 7 files; SKILL.md diff = `name:` + Process step 2; descriptions textually identical = arbitrary harness tie-break + double-counted listing budget). Kept `caveman-compress` canonical: it carries README.md (benchmarks table + caveman-toolkit branding) and SECURITY.md (Snyk false-positive writeup), and matches the caveman-* prefix invariant. Ported the better Process-step wording from `compress` into `caveman-compress` (`cd <directory_containing_this_SKILL.md>` instead of brittle `cd caveman-compress` which assumes cwd). Added `version: 1.0.0` to caveman-compress frontmatter (first versioned release; aligns with skill-versioning concept). Deleted: `skills/compress/`, `dist/compress.skill`, `~/.claude/skills/compress/` (manual prune — install.sh has no prune step; future `[install-ps1]` task should add `--prune` flag). Rebuilt + reinstalled `caveman-compress`. Slash-command impact: `/compress` removed; `/caveman-compress` + `/caveman:compress` (toolkit-canonical) remain. Concept page `concepts/compress-dedup.md` written (rationale + rejected alternatives: alias-stub has no harness mechanism; "keep both" wastes listing budget; "delete caveman-compress" loses README + SECURITY).
|
||||
|
||||
BIN
dist/caveman-compress.skill
vendored
BIN
dist/caveman-compress.skill
vendored
Binary file not shown.
BIN
dist/compress.skill
vendored
BIN
dist/compress.skill
vendored
Binary file not shown.
@@ -1,5 +1,6 @@
|
||||
---
|
||||
name: caveman-compress
|
||||
version: 1.0.0
|
||||
description: >
|
||||
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
|
||||
to save input tokens. Preserves all technical substance, code, URLs, and structure.
|
||||
@@ -19,11 +20,11 @@ Compress natural language files (CLAUDE.md, todos, preferences) into caveman-spe
|
||||
|
||||
## Process
|
||||
|
||||
1. The compression scripts live in `caveman-compress/scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `caveman-compress/scripts/__main__.py`.
|
||||
1. This SKILL.md lives alongside `scripts/` in the same directory. Find that directory.
|
||||
|
||||
2. Run:
|
||||
|
||||
cd caveman-compress && python3 -m scripts <absolute_filepath>
|
||||
cd <directory_containing_this_SKILL.md> && python3 -m scripts <absolute_filepath>
|
||||
|
||||
3. The CLI will:
|
||||
- detect file type (no tokens)
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
name: compress
|
||||
description: >
|
||||
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
|
||||
to save input tokens. Preserves all technical substance, code, URLs, and structure.
|
||||
Compressed version overwrites the original file. Human-readable backup saved as FILE.original.md.
|
||||
Trigger: /caveman:compress <filepath> or "compress memory file"
|
||||
---
|
||||
|
||||
# Caveman Compress
|
||||
|
||||
## Purpose
|
||||
|
||||
Compress natural language files (CLAUDE.md, todos, preferences) into caveman-speak to reduce input tokens. Compressed version overwrites original. Human-readable backup saved as `<filename>.original.md`.
|
||||
|
||||
## Trigger
|
||||
|
||||
`/caveman:compress <filepath>` or when user asks to compress a memory file.
|
||||
|
||||
## Process
|
||||
|
||||
1. This SKILL.md lives alongside `scripts/` in the same directory. Find that directory.
|
||||
|
||||
2. Run:
|
||||
|
||||
cd <directory_containing_this_SKILL.md> && python3 -m scripts <absolute_filepath>
|
||||
|
||||
3. The CLI will:
|
||||
- detect file type (no tokens)
|
||||
- call Claude to compress
|
||||
- validate output (no tokens)
|
||||
- if errors: cherry-pick fix with Claude (targeted fixes only, no recompression)
|
||||
- retry up to 2 times
|
||||
- if still failing after 2 retries: report error to user, leave original file untouched
|
||||
|
||||
4. Return result to user
|
||||
|
||||
## Compression Rules
|
||||
|
||||
### Remove
|
||||
- Articles: a, an, the
|
||||
- Filler: just, really, basically, actually, simply, essentially, generally
|
||||
- Pleasantries: "sure", "certainly", "of course", "happy to", "I'd recommend"
|
||||
- Hedging: "it might be worth", "you could consider", "it would be good to"
|
||||
- Redundant phrasing: "in order to" → "to", "make sure to" → "ensure", "the reason is because" → "because"
|
||||
- Connective fluff: "however", "furthermore", "additionally", "in addition"
|
||||
|
||||
### Preserve EXACTLY (never modify)
|
||||
- Code blocks (fenced ``` and indented)
|
||||
- Inline code (`backtick content`)
|
||||
- URLs and links (full URLs, markdown links)
|
||||
- File paths (`/src/components/...`, `./config.yaml`)
|
||||
- Commands (`npm install`, `git commit`, `docker build`)
|
||||
- Technical terms (library names, API names, protocols, algorithms)
|
||||
- Proper nouns (project names, people, companies)
|
||||
- Dates, version numbers, numeric values
|
||||
- Environment variables (`$HOME`, `NODE_ENV`)
|
||||
|
||||
### Preserve Structure
|
||||
- All markdown headings (keep exact heading text, compress body below)
|
||||
- Bullet point hierarchy (keep nesting level)
|
||||
- Numbered lists (keep numbering)
|
||||
- Tables (compress cell text, keep structure)
|
||||
- Frontmatter/YAML headers in markdown files
|
||||
|
||||
### Compress
|
||||
- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"
|
||||
- Fragments OK: "Run tests before commit" not "You should always run tests before committing"
|
||||
- Drop "you should", "make sure to", "remember to" — just state the action
|
||||
- Merge redundant bullets that say the same thing differently
|
||||
- Keep one example where multiple examples show the same pattern
|
||||
|
||||
CRITICAL RULE:
|
||||
Anything inside ``` ... ``` must be copied EXACTLY.
|
||||
Do not:
|
||||
- remove comments
|
||||
- remove spacing
|
||||
- reorder lines
|
||||
- shorten commands
|
||||
- simplify anything
|
||||
|
||||
Inline code (`...`) must be preserved EXACTLY.
|
||||
Do not modify anything inside backticks.
|
||||
|
||||
If file contains code blocks:
|
||||
- Treat code blocks as read-only regions
|
||||
- Only compress text outside them
|
||||
- Do not merge sections around code
|
||||
|
||||
## Pattern
|
||||
|
||||
Original:
|
||||
> You should always make sure to run the test suite before pushing any changes to the main branch. This is important because it helps catch bugs early and prevents broken builds from being deployed to production.
|
||||
|
||||
Compressed:
|
||||
> Run tests before push to main. Catch bugs early, prevent broken prod deploys.
|
||||
|
||||
Original:
|
||||
> The application uses a microservices architecture with the following components. The API gateway handles all incoming requests and routes them to the appropriate service. The authentication service is responsible for managing user sessions and JWT tokens.
|
||||
|
||||
Compressed:
|
||||
> Microservices architecture. API gateway route all requests to services. Auth service manage user sessions + JWT tokens.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- ONLY compress natural language files (.md, .txt, extensionless)
|
||||
- NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
|
||||
- If file has mixed content (prose + code), compress ONLY the prose sections
|
||||
- If unsure whether something is code or prose, leave it unchanged
|
||||
- Original file is backed up as FILE.original.md before overwriting
|
||||
- Never compress FILE.original.md (skip it)
|
||||
@@ -1,9 +0,0 @@
|
||||
"""Caveman compress scripts.
|
||||
|
||||
This package provides tools to compress natural language markdown files
|
||||
into caveman format to save input tokens.
|
||||
"""
|
||||
|
||||
__all__ = ["cli", "compress", "detect", "validate"]
|
||||
|
||||
__version__ = "1.0.0"
|
||||
@@ -1,3 +0,0 @@
|
||||
from .cli import main
|
||||
|
||||
main()
|
||||
@@ -1,78 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
# Support both direct execution and module import
|
||||
try:
|
||||
from .validate import validate
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from validate import validate
|
||||
|
||||
try:
|
||||
import tiktoken
|
||||
_enc = tiktoken.get_encoding("o200k_base")
|
||||
except ImportError:
|
||||
_enc = None
|
||||
|
||||
|
||||
def count_tokens(text):
|
||||
if _enc is None:
|
||||
return len(text.split()) # fallback: word count
|
||||
return len(_enc.encode(text))
|
||||
|
||||
|
||||
def benchmark_pair(orig_path: Path, comp_path: Path):
|
||||
orig_text = orig_path.read_text()
|
||||
comp_text = comp_path.read_text()
|
||||
|
||||
orig_tokens = count_tokens(orig_text)
|
||||
comp_tokens = count_tokens(comp_text)
|
||||
saved = 100 * (orig_tokens - comp_tokens) / orig_tokens if orig_tokens > 0 else 0.0
|
||||
result = validate(orig_path, comp_path)
|
||||
|
||||
return (comp_path.name, orig_tokens, comp_tokens, saved, result.is_valid)
|
||||
|
||||
|
||||
def print_table(rows):
|
||||
print("\n| File | Original | Compressed | Saved % | Valid |")
|
||||
print("|------|----------|------------|---------|-------|")
|
||||
for r in rows:
|
||||
print(f"| {r[0]} | {r[1]} | {r[2]} | {r[3]:.1f}% | {'✅' if r[4] else '❌'} |")
|
||||
|
||||
|
||||
def main():
|
||||
# Direct file pair: python3 benchmark.py original.md compressed.md
|
||||
if len(sys.argv) == 3:
|
||||
orig = Path(sys.argv[1]).resolve()
|
||||
comp = Path(sys.argv[2]).resolve()
|
||||
if not orig.exists():
|
||||
print(f"❌ Not found: {orig}")
|
||||
sys.exit(1)
|
||||
if not comp.exists():
|
||||
print(f"❌ Not found: {comp}")
|
||||
sys.exit(1)
|
||||
print_table([benchmark_pair(orig, comp)])
|
||||
return
|
||||
|
||||
# Glob mode: repo_root/tests/caveman-compress/
|
||||
tests_dir = Path(__file__).parent.parent.parent / "tests" / "caveman-compress"
|
||||
if not tests_dir.exists():
|
||||
print(f"❌ Tests dir not found: {tests_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
rows = []
|
||||
for orig in sorted(tests_dir.glob("*.original.md")):
|
||||
comp = orig.with_name(orig.stem.removesuffix(".original") + ".md")
|
||||
if comp.exists():
|
||||
rows.append(benchmark_pair(orig, comp))
|
||||
|
||||
if not rows:
|
||||
print("No compressed file pairs found.")
|
||||
return
|
||||
|
||||
print_table(rows)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,73 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Caveman Compress CLI
|
||||
|
||||
Usage:
|
||||
caveman <filepath>
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .compress import compress_file
|
||||
from .detect import detect_file_type, should_compress
|
||||
|
||||
|
||||
def print_usage():
|
||||
print("Usage: caveman <filepath>")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 2:
|
||||
print_usage()
|
||||
sys.exit(1)
|
||||
|
||||
filepath = Path(sys.argv[1])
|
||||
|
||||
# Check file exists
|
||||
if not filepath.exists():
|
||||
print(f"❌ File not found: {filepath}")
|
||||
sys.exit(1)
|
||||
|
||||
if not filepath.is_file():
|
||||
print(f"❌ Not a file: {filepath}")
|
||||
sys.exit(1)
|
||||
|
||||
filepath = filepath.resolve()
|
||||
|
||||
# Detect file type
|
||||
file_type = detect_file_type(filepath)
|
||||
|
||||
print(f"Detected: {file_type}")
|
||||
|
||||
# Check if compressible
|
||||
if not should_compress(filepath):
|
||||
print("Skipping: file is not natural language (code/config)")
|
||||
sys.exit(0)
|
||||
|
||||
print("Starting caveman compression...\n")
|
||||
|
||||
try:
|
||||
success = compress_file(filepath)
|
||||
|
||||
if success:
|
||||
print("\nCompression completed successfully")
|
||||
backup_path = filepath.with_name(filepath.stem + ".original.md")
|
||||
print(f"Compressed: {filepath}")
|
||||
print(f"Original: {backup_path}")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print("\n❌ Compression failed after retries")
|
||||
sys.exit(2)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user")
|
||||
sys.exit(130)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,227 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Caveman Memory Compression Orchestrator
|
||||
|
||||
Usage:
|
||||
python scripts/compress.py <filepath>
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
OUTER_FENCE_REGEX = re.compile(
|
||||
r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
|
||||
)
|
||||
|
||||
# Filenames and paths that almost certainly hold secrets or PII. Compressing
|
||||
# them ships raw bytes to the Anthropic API — a third-party data boundary that
|
||||
# developers on sensitive codebases cannot cross. detect.py already skips .env
|
||||
# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
|
||||
# slip through the natural-language filter. This is a hard refuse before read.
|
||||
SENSITIVE_BASENAME_REGEX = re.compile(
|
||||
r"(?ix)^("
|
||||
r"\.env(\..+)?"
|
||||
r"|\.netrc"
|
||||
r"|credentials(\..+)?"
|
||||
r"|secrets?(\..+)?"
|
||||
r"|passwords?(\..+)?"
|
||||
r"|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?"
|
||||
r"|authorized_keys"
|
||||
r"|known_hosts"
|
||||
r"|.*\.(pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg)"
|
||||
r")$"
|
||||
)
|
||||
|
||||
SENSITIVE_PATH_COMPONENTS = frozenset({".ssh", ".aws", ".gnupg", ".kube", ".docker"})
|
||||
|
||||
SENSITIVE_NAME_TOKENS = (
|
||||
"secret", "credential", "password", "passwd",
|
||||
"apikey", "accesskey", "token", "privatekey",
|
||||
)
|
||||
|
||||
|
||||
def is_sensitive_path(filepath: Path) -> bool:
|
||||
"""Heuristic denylist for files that must never be shipped to a third-party API."""
|
||||
name = filepath.name
|
||||
if SENSITIVE_BASENAME_REGEX.match(name):
|
||||
return True
|
||||
lowered_parts = {p.lower() for p in filepath.parts}
|
||||
if lowered_parts & SENSITIVE_PATH_COMPONENTS:
|
||||
return True
|
||||
# Normalize separators so "api-key" and "api_key" both match "apikey".
|
||||
lower = re.sub(r"[_\-\s.]", "", name.lower())
|
||||
return any(tok in lower for tok in SENSITIVE_NAME_TOKENS)
|
||||
|
||||
|
||||
def strip_llm_wrapper(text: str) -> str:
|
||||
"""Strip outer ```markdown ... ``` fence when it wraps the entire output."""
|
||||
m = OUTER_FENCE_REGEX.match(text)
|
||||
if m:
|
||||
return m.group(2)
|
||||
return text
|
||||
|
||||
from .detect import should_compress
|
||||
from .validate import validate
|
||||
|
||||
MAX_RETRIES = 2
|
||||
|
||||
|
||||
# ---------- Claude Calls ----------
|
||||
|
||||
|
||||
def call_claude(prompt: str) -> str:
|
||||
api_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
if api_key:
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic(api_key=api_key)
|
||||
msg = client.messages.create(
|
||||
model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),
|
||||
max_tokens=8192,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
return strip_llm_wrapper(msg.content[0].text.strip())
|
||||
except ImportError:
|
||||
pass # anthropic not installed, fall back to CLI
|
||||
# Fallback: use claude CLI (handles desktop auth)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["claude", "--print"],
|
||||
input=prompt,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
return strip_llm_wrapper(result.stdout.strip())
|
||||
except subprocess.CalledProcessError as e:
|
||||
raise RuntimeError(f"Claude call failed:\n{e.stderr}")
|
||||
|
||||
|
||||
def build_compress_prompt(original: str) -> str:
|
||||
return f"""
|
||||
Compress this markdown into caveman format.
|
||||
|
||||
STRICT RULES:
|
||||
- Do NOT modify anything inside ``` code blocks
|
||||
- Do NOT modify anything inside inline backticks
|
||||
- Preserve ALL URLs exactly
|
||||
- Preserve ALL headings exactly
|
||||
- Preserve file paths and commands
|
||||
- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.
|
||||
|
||||
Only compress natural language.
|
||||
|
||||
TEXT:
|
||||
{original}
|
||||
"""
|
||||
|
||||
|
||||
def build_fix_prompt(original: str, compressed: str, errors: List[str]) -> str:
|
||||
errors_str = "\n".join(f"- {e}" for e in errors)
|
||||
return f"""You are fixing a caveman-compressed markdown file. Specific validation errors were found.
|
||||
|
||||
CRITICAL RULES:
|
||||
- DO NOT recompress or rephrase the file
|
||||
- ONLY fix the listed errors — leave everything else exactly as-is
|
||||
- The ORIGINAL is provided as reference only (to restore missing content)
|
||||
- Preserve caveman style in all untouched sections
|
||||
|
||||
ERRORS TO FIX:
|
||||
{errors_str}
|
||||
|
||||
HOW TO FIX:
|
||||
- Missing URL: find it in ORIGINAL, restore it exactly where it belongs in COMPRESSED
|
||||
- Code block mismatch: find the exact code block in ORIGINAL, restore it in COMPRESSED
|
||||
- Heading mismatch: restore the exact heading text from ORIGINAL into COMPRESSED
|
||||
- Do not touch any section not mentioned in the errors
|
||||
|
||||
ORIGINAL (reference only):
|
||||
{original}
|
||||
|
||||
COMPRESSED (fix this):
|
||||
{compressed}
|
||||
|
||||
Return ONLY the fixed compressed file. No explanation.
|
||||
"""
|
||||
|
||||
|
||||
# ---------- Core Logic ----------
|
||||
|
||||
|
||||
def compress_file(filepath: Path) -> bool:
|
||||
# Resolve and validate path
|
||||
filepath = filepath.resolve()
|
||||
MAX_FILE_SIZE = 500_000 # 500KB
|
||||
if not filepath.exists():
|
||||
raise FileNotFoundError(f"File not found: {filepath}")
|
||||
if filepath.stat().st_size > MAX_FILE_SIZE:
|
||||
raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")
|
||||
|
||||
# Refuse files that look like they contain secrets or PII. Compressing ships
|
||||
# the raw bytes to the Anthropic API — a third-party boundary — so we fail
|
||||
# loudly rather than silently exfiltrate credentials or keys. Override is
|
||||
# intentional: the user must rename the file if the heuristic is wrong.
|
||||
if is_sensitive_path(filepath):
|
||||
raise ValueError(
|
||||
f"Refusing to compress {filepath}: filename looks sensitive "
|
||||
"(credentials, keys, secrets, or known private paths). "
|
||||
"Compression sends file contents to the Anthropic API. "
|
||||
"Rename the file if this is a false positive."
|
||||
)
|
||||
|
||||
print(f"Processing: {filepath}")
|
||||
|
||||
if not should_compress(filepath):
|
||||
print("Skipping (not natural language)")
|
||||
return False
|
||||
|
||||
original_text = filepath.read_text(errors="ignore")
|
||||
backup_path = filepath.with_name(filepath.stem + ".original.md")
|
||||
|
||||
# Check if backup already exists to prevent accidental overwriting
|
||||
if backup_path.exists():
|
||||
print(f"⚠️ Backup file already exists: {backup_path}")
|
||||
print("The original backup may contain important content.")
|
||||
print("Aborting to prevent data loss. Please remove or rename the backup file if you want to proceed.")
|
||||
return False
|
||||
|
||||
# Step 1: Compress
|
||||
print("Compressing with Claude...")
|
||||
compressed = call_claude(build_compress_prompt(original_text))
|
||||
|
||||
# Save original as backup, write compressed to original path
|
||||
backup_path.write_text(original_text)
|
||||
filepath.write_text(compressed)
|
||||
|
||||
# Step 2: Validate + Retry
|
||||
for attempt in range(MAX_RETRIES):
|
||||
print(f"\nValidation attempt {attempt + 1}")
|
||||
|
||||
result = validate(backup_path, filepath)
|
||||
|
||||
if result.is_valid:
|
||||
print("Validation passed")
|
||||
break
|
||||
|
||||
print("❌ Validation failed:")
|
||||
for err in result.errors:
|
||||
print(f" - {err}")
|
||||
|
||||
if attempt == MAX_RETRIES - 1:
|
||||
# Restore original on failure
|
||||
filepath.write_text(original_text)
|
||||
backup_path.unlink(missing_ok=True)
|
||||
print("❌ Failed after retries — original restored")
|
||||
return False
|
||||
|
||||
print("Fixing with Claude...")
|
||||
compressed = call_claude(
|
||||
build_fix_prompt(original_text, compressed, result.errors)
|
||||
)
|
||||
filepath.write_text(compressed)
|
||||
|
||||
return True
|
||||
@@ -1,121 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Detect whether a file is natural language (compressible) or code/config (skip)."""
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
# Extensions that are natural language and compressible
|
||||
COMPRESSIBLE_EXTENSIONS = {".md", ".txt", ".markdown", ".rst"}
|
||||
|
||||
# Extensions that are code/config and should be skipped
|
||||
SKIP_EXTENSIONS = {
|
||||
".py", ".js", ".ts", ".tsx", ".jsx", ".json", ".yaml", ".yml",
|
||||
".toml", ".env", ".lock", ".css", ".scss", ".html", ".xml",
|
||||
".sql", ".sh", ".bash", ".zsh", ".go", ".rs", ".java", ".c",
|
||||
".cpp", ".h", ".hpp", ".rb", ".php", ".swift", ".kt", ".lua",
|
||||
".dockerfile", ".makefile", ".csv", ".ini", ".cfg",
|
||||
}
|
||||
|
||||
# Patterns that indicate a line is code
|
||||
CODE_PATTERNS = [
|
||||
re.compile(r"^\s*(import |from .+ import |require\(|const |let |var )"),
|
||||
re.compile(r"^\s*(def |class |function |async function |export )"),
|
||||
re.compile(r"^\s*(if\s*\(|for\s*\(|while\s*\(|switch\s*\(|try\s*\{)"),
|
||||
re.compile(r"^\s*[\}\]\);]+\s*$"), # closing braces/brackets
|
||||
re.compile(r"^\s*@\w+"), # decorators/annotations
|
||||
re.compile(r'^\s*"[^"]+"\s*:\s*'), # JSON-like key-value
|
||||
re.compile(r"^\s*\w+\s*=\s*[{\[\(\"']"), # assignment with literal
|
||||
]
|
||||
|
||||
|
||||
def _is_code_line(line: str) -> bool:
|
||||
"""Check if a line looks like code."""
|
||||
return any(p.match(line) for p in CODE_PATTERNS)
|
||||
|
||||
|
||||
def _is_json_content(text: str) -> bool:
|
||||
"""Check if content is valid JSON."""
|
||||
try:
|
||||
json.loads(text)
|
||||
return True
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _is_yaml_content(lines: list[str]) -> bool:
|
||||
"""Heuristic: check if content looks like YAML."""
|
||||
yaml_indicators = 0
|
||||
for line in lines[:30]:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("---"):
|
||||
yaml_indicators += 1
|
||||
elif re.match(r"^\w[\w\s]*:\s", stripped):
|
||||
yaml_indicators += 1
|
||||
elif stripped.startswith("- ") and ":" in stripped:
|
||||
yaml_indicators += 1
|
||||
# If most non-empty lines look like YAML
|
||||
non_empty = sum(1 for l in lines[:30] if l.strip())
|
||||
return non_empty > 0 and yaml_indicators / non_empty > 0.6
|
||||
|
||||
|
||||
def detect_file_type(filepath: Path) -> str:
|
||||
"""Classify a file as 'natural_language', 'code', 'config', or 'unknown'.
|
||||
|
||||
Returns:
|
||||
One of: 'natural_language', 'code', 'config', 'unknown'
|
||||
"""
|
||||
ext = filepath.suffix.lower()
|
||||
|
||||
# Extension-based classification
|
||||
if ext in COMPRESSIBLE_EXTENSIONS:
|
||||
return "natural_language"
|
||||
if ext in SKIP_EXTENSIONS:
|
||||
return "code" if ext not in {".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".env"} else "config"
|
||||
|
||||
# Extensionless files (like CLAUDE.md, TODO) — check content
|
||||
if not ext:
|
||||
try:
|
||||
text = filepath.read_text(errors="ignore")
|
||||
except (OSError, PermissionError):
|
||||
return "unknown"
|
||||
|
||||
lines = text.splitlines()[:50]
|
||||
|
||||
if _is_json_content(text[:10000]):
|
||||
return "config"
|
||||
if _is_yaml_content(lines):
|
||||
return "config"
|
||||
|
||||
code_lines = sum(1 for l in lines if l.strip() and _is_code_line(l))
|
||||
non_empty = sum(1 for l in lines if l.strip())
|
||||
if non_empty > 0 and code_lines / non_empty > 0.4:
|
||||
return "code"
|
||||
|
||||
return "natural_language"
|
||||
|
||||
return "unknown"
|
||||
|
||||
|
||||
def should_compress(filepath: Path) -> bool:
|
||||
"""Return True if the file is natural language and should be compressed."""
|
||||
if not filepath.is_file():
|
||||
return False
|
||||
# Skip backup files
|
||||
if filepath.name.endswith(".original.md"):
|
||||
return False
|
||||
return detect_file_type(filepath) == "natural_language"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python detect.py <file1> [file2] ...")
|
||||
sys.exit(1)
|
||||
|
||||
for path_str in sys.argv[1:]:
|
||||
p = Path(path_str).resolve()
|
||||
file_type = detect_file_type(p)
|
||||
compress = should_compress(p)
|
||||
print(f" {p.name:30s} type={file_type:20s} compress={compress}")
|
||||
@@ -1,189 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
URL_REGEX = re.compile(r"https?://[^\s)]+")
|
||||
FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
|
||||
HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
|
||||
BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
|
||||
|
||||
# crude but effective path detection
|
||||
# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
|
||||
PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
|
||||
|
||||
|
||||
class ValidationResult:
|
||||
def __init__(self):
|
||||
self.is_valid = True
|
||||
self.errors = []
|
||||
self.warnings = []
|
||||
|
||||
def add_error(self, msg):
|
||||
self.is_valid = False
|
||||
self.errors.append(msg)
|
||||
|
||||
def add_warning(self, msg):
|
||||
self.warnings.append(msg)
|
||||
|
||||
|
||||
def read_file(path: Path) -> str:
|
||||
return path.read_text(errors="ignore")
|
||||
|
||||
|
||||
# ---------- Extractors ----------
|
||||
|
||||
|
||||
def extract_headings(text):
|
||||
return [(level, title.strip()) for level, title in HEADING_REGEX.findall(text)]
|
||||
|
||||
|
||||
def extract_code_blocks(text):
|
||||
"""Line-based fenced code block extractor.
|
||||
|
||||
Handles ``` and ~~~ fences with variable length (CommonMark: closing
|
||||
fence must use same char and be at least as long as opening). Supports
|
||||
nested fences (e.g. an outer 4-backtick block wrapping inner 3-backtick
|
||||
content).
|
||||
"""
|
||||
blocks = []
|
||||
lines = text.split("\n")
|
||||
i = 0
|
||||
n = len(lines)
|
||||
while i < n:
|
||||
m = FENCE_OPEN_REGEX.match(lines[i])
|
||||
if not m:
|
||||
i += 1
|
||||
continue
|
||||
fence_char = m.group(2)[0]
|
||||
fence_len = len(m.group(2))
|
||||
open_line = lines[i]
|
||||
block_lines = [open_line]
|
||||
i += 1
|
||||
closed = False
|
||||
while i < n:
|
||||
close_m = FENCE_OPEN_REGEX.match(lines[i])
|
||||
if (
|
||||
close_m
|
||||
and close_m.group(2)[0] == fence_char
|
||||
and len(close_m.group(2)) >= fence_len
|
||||
and close_m.group(3).strip() == ""
|
||||
):
|
||||
block_lines.append(lines[i])
|
||||
closed = True
|
||||
i += 1
|
||||
break
|
||||
block_lines.append(lines[i])
|
||||
i += 1
|
||||
if closed:
|
||||
blocks.append("\n".join(block_lines))
|
||||
# Unclosed fences are silently skipped — they indicate malformed markdown
|
||||
# and including them would cause false-positive validation failures.
|
||||
return blocks
|
||||
|
||||
|
||||
def extract_urls(text):
|
||||
return set(URL_REGEX.findall(text))
|
||||
|
||||
|
||||
def extract_paths(text):
|
||||
return set(PATH_REGEX.findall(text))
|
||||
|
||||
|
||||
def count_bullets(text):
|
||||
return len(BULLET_REGEX.findall(text))
|
||||
|
||||
|
||||
# ---------- Validators ----------
|
||||
|
||||
|
||||
def validate_headings(orig, comp, result):
|
||||
h1 = extract_headings(orig)
|
||||
h2 = extract_headings(comp)
|
||||
|
||||
if len(h1) != len(h2):
|
||||
result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
|
||||
|
||||
if h1 != h2:
|
||||
result.add_warning("Heading text/order changed")
|
||||
|
||||
|
||||
def validate_code_blocks(orig, comp, result):
|
||||
c1 = extract_code_blocks(orig)
|
||||
c2 = extract_code_blocks(comp)
|
||||
|
||||
if c1 != c2:
|
||||
result.add_error("Code blocks not preserved exactly")
|
||||
|
||||
|
||||
def validate_urls(orig, comp, result):
|
||||
u1 = extract_urls(orig)
|
||||
u2 = extract_urls(comp)
|
||||
|
||||
if u1 != u2:
|
||||
result.add_error(f"URL mismatch: lost={u1 - u2}, added={u2 - u1}")
|
||||
|
||||
|
||||
def validate_paths(orig, comp, result):
|
||||
p1 = extract_paths(orig)
|
||||
p2 = extract_paths(comp)
|
||||
|
||||
if p1 != p2:
|
||||
result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
|
||||
|
||||
|
||||
def validate_bullets(orig, comp, result):
|
||||
b1 = count_bullets(orig)
|
||||
b2 = count_bullets(comp)
|
||||
|
||||
if b1 == 0:
|
||||
return
|
||||
|
||||
diff = abs(b1 - b2) / b1
|
||||
|
||||
if diff > 0.15:
|
||||
result.add_warning(f"Bullet count changed too much: {b1} -> {b2}")
|
||||
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
|
||||
def validate(original_path: Path, compressed_path: Path) -> ValidationResult:
|
||||
result = ValidationResult()
|
||||
|
||||
orig = read_file(original_path)
|
||||
comp = read_file(compressed_path)
|
||||
|
||||
validate_headings(orig, comp, result)
|
||||
validate_code_blocks(orig, comp, result)
|
||||
validate_urls(orig, comp, result)
|
||||
validate_paths(orig, comp, result)
|
||||
validate_bullets(orig, comp, result)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------- CLI ----------
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) != 3:
|
||||
print("Usage: python validate.py <original> <compressed>")
|
||||
sys.exit(1)
|
||||
|
||||
orig = Path(sys.argv[1]).resolve()
|
||||
comp = Path(sys.argv[2]).resolve()
|
||||
|
||||
res = validate(orig, comp)
|
||||
|
||||
print(f"\nValid: {res.is_valid}")
|
||||
|
||||
if res.errors:
|
||||
print("\nErrors:")
|
||||
for e in res.errors:
|
||||
print(f" - {e}")
|
||||
|
||||
if res.warnings:
|
||||
print("\nWarnings:")
|
||||
for w in res.warnings:
|
||||
print(f" - {w}")
|
||||
Reference in New Issue
Block a user