Skip to content

ci: add manifest-drift gate + correct stale generated artifacts#103

Merged
Timer merged 2 commits into
mainfrom
manifest-ci-gate
Jun 26, 2026
Merged

ci: add manifest-drift gate + correct stale generated artifacts#103
Timer merged 2 commits into
mainfrom
manifest-ci-gate

Conversation

@Timer

@Timer Timer commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a CI gate that fails when the committed generated/skill-manifest.json drifts from its skill sources, and heals the existing drift it surfaced.

The bug this prevents

The committed manifest had silently drifted from source:

  • 498c263 (fixing strict workflow skill detection, Apr 30) deliberately removed 11 over-broad workflow path patterns (**/chain-engine*, **/orchestrat*, **/escalation*, **/state-machine*, **/pipeline-engine*, …) from skills/workflow/SKILL.md and the manifest.
  • 5641bff (feat(skills): add vercel-connect skill #94, Jun 9, regenerate skill-manifest with vercel-connect) reintroduced them into the manifest only — it regenerated on a branch that predated the Apr 30 removal, so a stale artifact got committed. At that commit skills/workflow/SKILL.md had zero occurrences of those patterns.

Root cause: CI runs bun run build (which regenerates the manifest into the working tree) and then tests against that fresh copy. It never checks the committed manifest against source, so a stale commit sails through. The manifest also embeds a volatile generatedAt timestamp, so a plain git diff --exit-code can't be used.

Fix

  • scripts/build-manifest.ts --check — rebuilds in-memory, compares to the committed file (normalizing generatedAt), exits non-zero on drift. Exposed as bun run build:manifest:check.
  • CI — runs build:manifest:check and the existing build:from-skills:check before bun run build, so they validate committed artifacts on the pristine tree instead of after the build clobbers them.
  • Corrections — regenerated skill-manifest.json (workflow patterns) and skill-catalog.md (workflow patterns + a stale vercel-connect entry that had also drifted).

Verification

  • bun run build:manifest:check → passes on the healed manifest; fails (exit 1) when a stale pattern is reintroduced (tested).
  • bun run validate → passes (only the pre-existing workflow chainTo warning).
  • bun test878 pass, 0 fail.

🤖 Generated with Claude Code


Note

Low Risk
Changes are CI ordering plus regenerated docs/catalog metadata; they tighten correctness of skill routing patterns without touching runtime app code.

Overview
CI now fails when committed generated artifacts drift from skill sources, by running bun run build:manifest:check and moving build:from-skills:check before bun run build so checks run on the pristine checkout instead of after the build overwrites regenerated files.

generated/skill-catalog.md is refreshed to match current sources: workflow drops the reintroduced broad path globs (back to 6 path patterns), and vercel-connect reflects current path/import detection (agent/connections/**, agent/channels/**, @vercel/connect/eve).

Reviewed by Cursor Bugbot for commit 5a55c8f. Bugbot is set up for automated code reviews on this repo. Configure here.

The committed skill-manifest.json had drifted from its source: the
`workflow` skill carried 11 path patterns (**/chain-engine*, **/orchestrat*,
**/escalation*, etc.) that were deliberately removed from skills/workflow/
SKILL.md in 498c263 ("fixing strict workflow skill detection"). They were
silently reintroduced into the manifest in 5641bff (#94), which regenerated
the manifest on a branch predating that removal. The skill-catalog.md had
the same workflow drift plus a stale vercel-connect entry.

Root cause: CI runs `bun run build` (which regenerates the manifest into the
working tree) and then tests against that fresh copy — it never verifies the
*committed* manifest matches source, so a stale committed manifest passes.

Fix:
- Add `--check` to scripts/build-manifest.ts: rebuilds in-memory and compares
  to the committed file (normalizing the volatile generatedAt timestamp),
  exiting non-zero on drift. Exposed as `bun run build:manifest:check`.
- Wire build:manifest:check (and the existing build:from-skills:check) into
  CI *before* the build step, so they validate the committed artifacts on the
  pristine tree rather than after build clobbers them.
- Regenerate skill-manifest.json and skill-catalog.md to heal the drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…forms

scanSkillsDir() iterated readdirSync() output directly, whose order is
filesystem-dependent — roughly alphabetical on macOS/APFS but arbitrary on
Linux/ext4. The committed manifest was therefore generated in macOS order,
and build:manifest:check failed on the Linux CI runner because a fresh build
produced the same skills in a different key order (CI run on #103).

Sort the entries so manifest/catalog generation is byte-identical on every
platform. Regenerates skill-manifest.json and skill-catalog.md into sorted
order (one-time reordering churn; no semantic change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Timer

Timer commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up commit 5a55c8f: the first CI run surfaced a real cross-platform bug — scanSkillsDir() iterated readdirSync() directly, whose order is filesystem-dependent (alphabetical-ish on macOS/APFS, arbitrary on Linux/ext4). So the macOS-generated committed manifest didn't match a fresh Linux build and build:manifest:check failed on the runner. Fixed by sorting the scan, which makes manifest/catalog generation byte-identical on every platform (one-time reordering churn, no semantic change). CI is now green. ✅

This is a nice validation that the gate works — it caught non-determinism that was previously invisible.

@Timer Timer enabled auto-merge (squash) June 26, 2026 19:08
@Timer Timer merged commit 3b6cbeb into main Jun 26, 2026
4 checks passed
@Timer Timer deleted the manifest-ci-gate branch June 26, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants