Green before it was tested: scoping a CI matrix without guessing

A pull request shows a column of green ticks. Every one of them really ran, really passed, and not one of them touched the code in the diff. That is not a hypothetical. It is the ordinary outcome of scoping a test matrix by hand.

The window

In a repository holding several services, running every service’s quality and test jobs on every pull request is the honest default. It is also the slow one. As the matrix grows the jobs queue, and the queue opens a gap: the cheap governance jobs finish in a minute and report green while the job that would actually exercise the changed code has not been created yet. Anyone reading the tick list during that gap sees a green pull request. Merge inside the window and the change lands untested.

The obvious fix is to run less. That is where the lie enters. Scoping done by hand — a path filter someone wrote once, a matrix entry commented out to unblock a release — narrows the matrix from a memory of the dependency graph rather than from the graph. It works until a shared file moves, and then the pull request is green for exactly the reason it was green before: nothing that could have failed was asked to run.

The rule

The fix is not smarter narrowing. It is an explicit rule about what happens when the tool is unsure.

A change confined to one service narrows the matrix. Everything else runs everything.

That asymmetry is the design, not a shortcoming. The failure being prevented is a false green, so widening wrongly costs a few minutes of compute and narrowing wrongly costs a merged regression nobody saw. Those are not comparable, so the fallback is never “guess” — it is “run everything.”

Concretely, one function takes the list of changed paths and returns either a set of services or the word ALL:

• a file inside one service’s directory → that service, plus every service that imports it
• any shared or root path — the workflow file, the build entry point, an instruction contract → everything
• an empty diff → everything, because no information is not the same as no impact
• a service directory that no longer exists on this ref → everything; a deletion or rename breaks whatever depended on it, and no per-service job would be created to notice
• a pointer bump for a nested repository → scoped exactly like an in-tree change to that component, because version control reports it as the bare directory and it would otherwise fall through to the widen-all branch
• a short, explicitly named list of paths proven to be read by nothing in CI → nothing at all

That last one is the only place the tool is allowed to be clever, and it is deliberately a list of filenames rather than a wildcard. A blanket rule over a directory would silently make some future CI-wired script in that directory inert. Each entry carries a comment recording the command that proved it unread.

The map is the dangerous part

The reverse-dependency edges are hand-maintained, and a hand-maintained graph rots. A cross-service import added without a matching edge does not fail loudly; it quietly drops coverage. The dependent stops running when its dependency changes — the same false green wearing a different hat.

So the map defends itself. Before it scopes anything, the script greps every tracked service file for imports of each package it knows about and fails the run when an importer is missing from the map. An importer that has no CI jobs of its own only warns, because there is nothing actionable to scope, but the warning names it as a coverage gap rather than swallowing it.

A second assertion covers an edge no import grep can see. One shared store is reached at runtime through an environment variable and a path injection rather than a normal import, so that edge is verified by grepping for the runtime markers instead.

A third assertion reads the workflow file itself. The workflow carries its own hand-written copy of the same graph — the sibling packages installed before each service’s tests run — so the script parses those blocks out, requires every one to be backed by an edge in the map, and requires the copies in different jobs to agree with each other. Two hand-maintained copies of one graph is a drift bug with a date on it; the gate turns it into a red build the day they disagree.

One gate allowed to narrow, one not

A second, independent decision rides along: which services had their dependency declarations change, so the vulnerability scan only re-scans where dependencies actually moved. It applies the same conservatism from the other direction. It narrows on pull requests only — scheduled and manually triggered runs still scan everything — and when the base commit for the diff is unavailable it emits every service rather than none. A security gate is precisely the thing that must not narrow on missing information.

It ships with its own tests

The unusual part, for a CI shell script, is that it has a test harness. The logic lives in one file rather than inline in the workflow, and the harness sources that file rather than copying it, so the harness cannot drift from what CI executes. It runs as a CI step before scoping and takes seconds.

That is only possible because the decision function is pure. It takes a newline-separated list of paths, prints a decision, and touches nothing else — no file writes, no environment mutation, no network. The side effects, writing the output variables and printing the run summary, live in separate functions. A pure function at the centre is what makes a shell script testable at all. The cases that depend on the filesystem — a deleted directory, an excluded directory, an existing directory that is not a service — run against a temporary fixture tree rather than the real repository, and the drift assertions run against throwaway repositories seeded with a deliberately rogue import.

The harness spends most of its effort on negatives, which is where the value is: that the two CI-wired scripts still widen to everything when they change, because they are the scope logic; that a loose markdown file sitting directly under the services directory is read as prose and not as a service name; that a documentation-only edit does not suppress a genuine shared-path widen when both land in the same diff.

Say what you skipped

Whatever the decision, the run prints both halves: what ran, and what was skipped and why. Silent narrowing is how the window reopens — someone reads a green pull request, has no idea a matrix was trimmed underneath them, and merges. If the tool is going to run less, it has to say so where the person merging will read it.

Why there is no repository link here

This one is staying private, and for the honest reason. The reverse-dependency map hardcodes the names of internal services; strip them out and what is left is a shape, not a tool. And the general problem — derive the affected target set from a real dependency graph — is already solved properly by mature build systems that read the graph out of the build files instead of a hand-written table. Publishing this would be offering a worse version of something that exists.

What generalises is smaller and needs no repository: name the failure you are preventing, then make the fallback the safe side of it. If the failure is a false green, the fallback is run everything, and every branch you add has to earn its narrowing with a comment showing the command that proved the path inert. The rest is bookkeeping — and, as usual, behavioural rules are suggestions; structure is a constraint.