A skill file is production config. It is also prose.

The flow

Four steps. Only step three involves an agent, which keeps the surface where behaviour can vary as small as possible.

The four outcomes

Three come from the agent's judgement. The fourth comes from the agent failing to produce a judgement at all, which is treated as a distinct condition on purpose.

What a developer sees

One check named Review changed skills, and one comment. The comment is upserted against a hidden marker, so pushing a fix edits the existing comment rather than stacking a new one underneath. Findings are grouped by severity, and each carries the rule ID, the file and line, the offending text quoted, what it causes, and the fix.

The label tracks current state rather than history. It is added on WARN or BLOCK and removed on PASS, so a fixed pull request does not sit there wearing a stale needs-revision tag.

Every run uploads verdict.txt, the raw NDJSON stream, the rendered comment, and the computed file list as an artifact, retained for fourteen days. When the gate errors, that artifact is the only place the agent's actual output survives.

The files

Nine files, with a deliberate division of labour: the prompt orchestrates, the rubric judges, and the scripts only extract and map. No script contains an opinion about skill quality, and the rubric contains no code.

skill-check/
  README.md                      setup and adoption
  skills/                        WATCHED. Business-authored agent skills.
    weekly-revenue-summary/
      SKILL.md                   example: a skill a business team owns
      references/                its business definitions
  evaluator/                     the reviewer, deliberately outside skills/
    SKILL.md                     review workflow and output contract
    references/rubric.md         every rule, severity, tier derivation
  prompts/evaluate.md            the prompt passed to cortex exec
  scripts/
    changed-skills.sh            PR diff to changed SKILL.md paths
    parse-verdict.sh             verdict to exit code
    parse_verdict.py             parsing, validation, comment rendering
    auth-pat-fallback.sh         connections.toml from a PAT, fallback only
  fixtures/{pass,warn,block}/    sample skills producing each tier
.github/workflows/
  skill-check.yml                the workflow, must be at repo root

How the agent is invoked

The action installs and pins the CLI. It is not used to run the prompt, because its prompt-args input is split on whitespace, which makes any argument containing a space unreachable, and it hardcodes flags this gate needs to control. Calling cortex exec directly in a run: step is also what the action's own documentation demonstrates.

cortex exec \
  --file skill-check/prompts/evaluate.md \
  --connection "$CONNECTION_NAME" \
  --allowed "read,grep,glob" \
  --no-mcp \
  --no-history \
  --bypass \
  --max-turns 40 \
  --output-last-message verdict.txt \
  --format json > cortex-stream.ndjson 2>&1 || true

The verdict contract

This is the entire interface between the agent and the workflow. The agent ends its response with one fenced JSON block, and the gate reads the last such block in the file.

{
  "tier": "BLOCK",
  "summary": "one sentence naming the most serious problem",
  "files_reviewed": ["skill-check/skills/weekly-revenue-summary/SKILL.md"],
  "findings": [
    {
      "severity": "critical",
      "rule": "fm-description-required",
      "file": "skill-check/skills/weekly-revenue-summary/SKILL.md",
      "line": 2,
      "quote": "name: weekly-revenue-summary",
      "issue": "Frontmatter has no description, so the skill can never be triggered.",
      "fix": "Add a description stating what it does and when to use it, including the words a user would say: revenue, weekly report, week over week."
    }
  ]
}

Two fields carry most of the weight. quote makes a finding locatable, and a finding the author cannot locate is not actionable. fix has to be specific enough to apply without further thought, which is why the rubric rejects "Improve the description" as a fix and requires the actual words to add.

The gate checks the agent's arithmetic

The tier is derived from severity counts, so it can be recomputed. parse_verdict.py does exactly that and escalates when the agent under-reported:

# findings say BLOCK, the agent labelled it PASS
WARNING: agent reported tier PASS but its findings derive BLOCK
         (1 critical, 0 warning); escalating to BLOCK

An agent that lists a critical finding and then labels the verdict PASS has contradicted itself. The findings are evidence and the tier is a claim about that evidence, so the evidence wins. Escalation only ever runs in the strict direction: the gate will raise a tier the agent under-called, never lower one it over-called.

One file owns all the judgement

evaluator/references/rubric.md defines every rule, every severity, and the tier derivation. The scripts only extract and map, so making the gate stricter or looser is a documentation change that a reviewer can read in a diff.

Severity

Severity is a property of the rule, not a judgement about how bad a particular instance feels. That distinction is what keeps verdicts consistent between runs.

Tier derivation

Applied in order, stopping at the first match:

1. One or more critical findings, tier is BLOCK.
2. Three or more warning findings, tier is BLOCK.
3. One or two warning findings, tier is WARN.
4. Only suggestion findings, or none, tier is PASS.

Rule two is the opinionated one. Volume of unaddressed warnings is treated as a defect in its own right, on the grounds that a file with five warnings is not a working file with minor notes, it is a file nobody has maintained. Raise the threshold if that is too aggressive for your team.

Mechanical rules

These have single defensible answers. Two reviewers looking at the same file should agree.

Judgement rules

These require reading the file end to end, and they are why the reviewer is an agent. The rubric constrains them in one important way: a finding is only reportable if the reviewer can quote the specific text that violates the rule. No quote, no finding.

Adopting it

The template repository carries the exact commands, in skill-check/README.md. This is the shape of the work and the places it usually goes wrong.

Authentication

OIDC workload identity federation is the default and the better option, because it mints a short-lived token per run and leaves no long-lived secret in the repository to leak or rotate.

CREATE USER skill_check_ci
  WORKLOAD_IDENTITY = (
    TYPE = OIDC
    ISSUER = 'https://token.actions.githubusercontent.com'
    SUBJECT = 'repo:YOUR_ORG/YOUR_REPO:pull_request'
  )
  TYPE = SERVICE
  DEFAULT_ROLE = SKILL_CHECK_CI;

If OIDC cannot be enabled yet, set the repository variable SKILL_CHECK_AUTH_MODE to pat and add a SNOWFLAKE_PAT secret. The fallback writes the token to its own file at mode 600 and references it by token_file_path rather than inlining it into the TOML, and it refuses to overwrite an existing connection so it cannot silently downgrade OIDC credentials to a long-lived secret. Treat it as a starting point rather than a destination.

Running it locally

Three fixtures ship with the template, one per tier, so the gate can be exercised without a pull request or a customer repository:

# point the file list at a fixture, then run the gate
echo "skill-check/fixtures/block/data-loader/SKILL.md" > skill-check/.changed-skills.txt

cortex exec --file skill-check/prompts/evaluate.md \
  --allowed "read,grep,glob" --no-mcp --no-history --bypass \
  --max-turns 40 --output-last-message verdict.txt

VERDICT_FILE=verdict.txt bash skill-check/scripts/parse-verdict.sh

Verified results from the committed fixtures:

Test the failure paths, not just the happy one

A gate that always passes is indistinguishable from a gate that works, so the checks worth running are the ones that prove it can say no:

• Feed parse-verdict.sh a verdict file containing malformed JSON and confirm it reports a gate error rather than a pass. A silent pass on a parse failure makes the entire gate decorative.
• Prompt the reviewer to write a file and confirm the tool call is denied. That proves the allowlist is doing something.
• Run changed-skills.sh with a base commit that is not present and confirm it fails loudly. On a shallow clone the diff silently returns nothing, which would pass a pull request that changed every skill in the repository. This is why the workflow sets fetch-depth: 0.

Exit status is not a verdict

This is the single most important design decision, and the reasoning generalises to any agent running in CI. There are two independent questions, and they are easy to conflate:

They come apart in practice. A run that answers correctly can still report an error, because a turn ceiling is a property of the run rather than of the answer:

# the agent answered correctly, and still reported an error
{"type":"assistant","message":{"content":[{"type":"text","text":"PONG"}]}}
{"type":"result","subtype":"error_max_turns","is_error":true}

Reading exit status as the verdict breaks in both directions. A healthy pull request gets blocked because a ceiling was hit or a network call blipped, and the author has no idea why since nothing is wrong with their file. Worse, a run that legitimately concluded BLOCK but exited cleanly reads as a pass, and the gate merges a file the agent just called broken. That second failure is silent, and a gate that always passes looks exactly like a gate that works.

So the verdict comes from a file the agent writes, and the absence of a verdict is its own outcome rather than an approval.

Why the JSON stream is not parsed

--format json emits NDJSON with human-readable status lines interleaved:

✓ Auto-apply [_CORTEX_CODE_DEFAULT]: 5 skill repos
History redaction mode is enabled. Session history will not be persisted.
{"type":"system","subtype":"init", ...}
{"type":"assistant","message": ...}
Max tool call iterations reached
{"type":"result","subtype":"error_max_turns","is_error":true, ...}

Piping that into jq fails on the first line. You can filter to the lines that parse, but then the gate depends on which status messages this CLI version happens to print and on the shape of the envelope, both of which change between versions. --output-last-message writes only the final message, so the gate is coupled to the verdict contract rather than to the CLI's output format. The stream is still captured as a diagnostic artifact, which is the right role for it.

Why the CLI version is not pinned

A stale CLI refuses to run headlessly at all:

$ cortex exec "..." --format json
This version is no longer supported please run `cortex update`

The instinct is to pin a version so runs are reproducible. That instinct is wrong here, and this gate learned it the hard way on its own first pull request. A pin does not prevent that failure, it schedules it: the pinned build eventually becomes the build the CLI refuses to run, and then every pull request fails at once for a reason unrelated to any skill file. The workflow uses latest, which cannot go stale.

If you do need strict reproducibility, pin the full build-qualified version rather than a bare semver. A bare version is rejected:

$ cortex update 1.1.53
Invalid version format: 1.1.53

# the accepted form is what `cortex versions` prints
$ cortex update 1.1.53+181504.3a965a854064

The reproducibility a pin buys is smaller than it looks. Bundled skills load on every run regardless of configuration and there is no flag to suppress them, so a pin freezes that set, but the rubric this gate judges against is read from the repository by explicit path. Verdicts are anchored to a file you version, not to whatever shipped with the CLI.

A flag that was removed

An earlier version passed --skills at a committed skills.json, intending to load only the evaluator skill. It does not work: with the flag set, the agent reported the skill as not installed and fell back to reading the files directly. The flag is also placement-sensitive. Passed as --skills value after exec the value is consumed as a positional message and collides with --file; placed before exec it prints help; only --skills=value parses at all.

Rather than depend on that, the prompt reads SKILL.md and references/rubric.md by explicit path. That is deterministic, needs no registration step, and removes a dependency on undocumented behaviour. The evaluator is still authored as a skill, because that is the right shape for the content and it can be loaded interactively when iterating on the rubric.

Adapting the pattern

Most of this template is not about skills. To gate a different kind of change, four things move and the rest stays:

Unchanged: the verdict contract, the tier-to-exit-code mapping, the escalation cross-check, the comment upsert, the label sync, and the gate-error path. Those are domain-independent and are the parts that took the longest to get right.