ci: add manifest-drift gate + correct stale generated artifacts#103
Merged
Conversation
The committed skill-manifest.json had drifted from its source: the `workflow` skill carried 11 path patterns (**/chain-engine*, **/orchestrat*, **/escalation*, etc.) that were deliberately removed from skills/workflow/ SKILL.md in 498c263 ("fixing strict workflow skill detection"). They were silently reintroduced into the manifest in 5641bff (#94), which regenerated the manifest on a branch predating that removal. The skill-catalog.md had the same workflow drift plus a stale vercel-connect entry. Root cause: CI runs `bun run build` (which regenerates the manifest into the working tree) and then tests against that fresh copy — it never verifies the *committed* manifest matches source, so a stale committed manifest passes. Fix: - Add `--check` to scripts/build-manifest.ts: rebuilds in-memory and compares to the committed file (normalizing the volatile generatedAt timestamp), exiting non-zero on drift. Exposed as `bun run build:manifest:check`. - Wire build:manifest:check (and the existing build:from-skills:check) into CI *before* the build step, so they validate the committed artifacts on the pristine tree rather than after build clobbers them. - Regenerate skill-manifest.json and skill-catalog.md to heal the drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…forms scanSkillsDir() iterated readdirSync() output directly, whose order is filesystem-dependent — roughly alphabetical on macOS/APFS but arbitrary on Linux/ext4. The committed manifest was therefore generated in macOS order, and build:manifest:check failed on the Linux CI runner because a fresh build produced the same skills in a different key order (CI run on #103). Sort the entries so manifest/catalog generation is byte-identical on every platform. Regenerates skill-manifest.json and skill-catalog.md into sorted order (one-time reordering churn; no semantic change). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Follow-up commit This is a nice validation that the gate works — it caught non-determinism that was previously invisible. |
Melkeydev
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a CI gate that fails when the committed
generated/skill-manifest.jsondrifts from its skill sources, and heals the existing drift it surfaced.The bug this prevents
The committed manifest had silently drifted from source:
fixing strict workflow skill detection, Apr 30) deliberately removed 11 over-broadworkflowpath patterns (**/chain-engine*,**/orchestrat*,**/escalation*,**/state-machine*,**/pipeline-engine*, …) fromskills/workflow/SKILL.mdand the manifest.regenerate skill-manifest with vercel-connect) reintroduced them into the manifest only — it regenerated on a branch that predated the Apr 30 removal, so a stale artifact got committed. At that commitskills/workflow/SKILL.mdhad zero occurrences of those patterns.Root cause: CI runs
bun run build(which regenerates the manifest into the working tree) and then tests against that fresh copy. It never checks the committed manifest against source, so a stale commit sails through. The manifest also embeds a volatilegeneratedAttimestamp, so a plaingit diff --exit-codecan't be used.Fix
scripts/build-manifest.ts --check— rebuilds in-memory, compares to the committed file (normalizinggeneratedAt), exits non-zero on drift. Exposed asbun run build:manifest:check.build:manifest:checkand the existingbuild:from-skills:checkbeforebun run build, so they validate committed artifacts on the pristine tree instead of after the build clobbers them.skill-manifest.json(workflow patterns) andskill-catalog.md(workflow patterns + a stalevercel-connectentry that had also drifted).Verification
bun run build:manifest:check→ passes on the healed manifest; fails (exit 1) when a stale pattern is reintroduced (tested).bun run validate→ passes (only the pre-existingworkflowchainTo warning).bun test→ 878 pass, 0 fail.🤖 Generated with Claude Code
Note
Low Risk
Changes are CI ordering plus regenerated docs/catalog metadata; they tighten correctness of skill routing patterns without touching runtime app code.
Overview
CI now fails when committed generated artifacts drift from skill sources, by running
bun run build:manifest:checkand movingbuild:from-skills:checkbeforebun run buildso checks run on the pristine checkout instead of after the build overwrites regenerated files.generated/skill-catalog.mdis refreshed to match current sources:workflowdrops the reintroduced broad path globs (back to 6 path patterns), andvercel-connectreflects current path/import detection (agent/connections/**,agent/channels/**,@vercel/connect/eve).Reviewed by Cursor Bugbot for commit 5a55c8f. Bugbot is set up for automated code reviews on this repo. Configure here.