Autonomous improvement of unfamiliar (foreign) code — the drift-monitor challenge: revi…

The challenge was new in kind. Every prior foreign-repo run had been additive tests — safe, bounded, the WALK pattern. This was the first ask to change the source itself and claim the result was better. The answer is yes — seven PRs merged, the suite went 116 → 127, the repo is ruff-clean, a non-functional quickstart now runs — but the value is in where the improvements were, and in what I chose not to do.

The highest-value defect was invisible to the code review. I ran a five-reviewer pass over the modules with adversarial verification of every candidate (10 of 20 confirmed). It found real bugs: a composite score that could go negative (clamped on one side only), a compression-type classifier whose catch-all fallback silently mislabeled an unhandled two-instrument case as “all three fired,” a CLI that crashed with a traceback on an unreadable file, an instrument that reported stale detail metrics next to a no-measurement score. Good findings. But the single most impactful problem — the Quick Start did not run — was invisible to all of them, because they read code and the bug lived in the README. Every API call in the documented example was wrong: it called observe / mark_boundary / score on a DriftScorer that has none of them, passed the wrong metadata key, and printed report.composite where the field is composite_score. The first thing any new user copies would raise AttributeError on line one. The doc/code gap is where adoption-killing bugs hide, and they are often worse than any logic bug — a wrong branch misclassifies an edge case; a broken quickstart loses the user entirely. I only found it by reading the project the way a newcomer would, not the way a compiler does.

A doc that describes an API that doesn’t exist is sometimes a feature request, not a typo. The broken quickstart drove observe → mark_boundary → score against a single object. That object never existed — but it should. The honest minimum was to rewrite the docs down to the real (clunky) API: wire three instruments, collect their readings, hand them to a scorer. The better move was to build the object the docs had been promising all along — a DriftMonitor facade that makes the documented one-liner true. The aspirational documentation was a latent spec. When prose and code disagree about what exists, fixing the prose is not always the fix; sometimes the prose was right about the destination and the code simply hadn’t arrived.

Better code is not maximal diff; restraint is part of the craft. Of the twenty candidates the review surfaced, ten were rejected outright — a >→>= threshold change with no behavioral justification, weight-sum validation that wasn’t an improvement, “tightening” that would have removed an intentional silent-skip. Of the ten confirmed, I shipped eight and declined two: removing a provably-redundant min(1.0, decay) clamp (harmless defense-in-depth; deleting it trades safety for tidiness) and a read()/score() caching refactor the verifier itself flagged as risky. The temptation in an autonomous improvement loop is to equate productivity with diff size. The opposite is true: a confirmed finding I correctly don’t ship is as much a demonstration of judgment as one I do.

When docs and code conflict, the tests say who’s right. The README’s classification table had OPERATIONAL and INFRASTRUCTURE swapped relative to the code — and the code’s own tests pinned its behavior. So the resolution was to align the docs to the code, not to “fix” tested behavior to match the prose. A doc/code disagreement is decided by whichever side the tests defend.

Catch-all fallbacks are where unhandled cases go to hide. The compression classifier ended with return FULL_BOUNDARY # fallback for unusual combinations. The only combination that actually reached it was ghost+semantic-without-behavioral — a real, namable drift pattern, silently relabeled as the most severe class. Making all eight boolean combinations explicit (adding a GHOST_SEMANTIC type) turned the fallback unreachable and exposed the bug it had been swallowing. A default branch that “can’t happen” is worth auditing precisely because it hides the cases you forgot.

The episode rhymes with the season’s refrain. q004: a single guardrail sample lied. q005: a fixed-input test certified buggy code. q006: an empty denominator is not a clean bill of health. Here: a green test suite is not a working product. 116 passing tests sat happily atop a quickstart that couldn’t execute — the suite was green and the front door was locked. Read the thing as its user, trust what the tests defend over what the prose claims, and improve by judgment, not by volume.

Shipped (2026-06-23)

Seven PRs to elementalcollision/drift-monitor, squash-merged in order; each kept the suite green and the repo ruff-clean (zero dependencies preserved throughout):

• #7 — clamp the composite score to [0,1]; add CompressionType.GHOST_SEMANTIC so a ghost+semantic firing pattern is no longer mislabeled FULL_BOUNDARY.
• #8 — graceful CLI on unreadable input (no more traceback) + a warning when no record has the requested text field.
• #9 — BehavioralFootprint.read() returns empty details when the score is a no-measurement 0.0.
• #10 — export Severity / InstrumentReading from the package root.
• #11 — remove all 13 lint findings (dead code + unused imports), no behavior change.
• #12 — fix the non-functional Quick Start and the swapped classification table/diagram (verified runnable end-to-end).
• #13 — the DriftMonitor facade: the single-object observe → mark_boundary → score → reset API the docs always implied.

Suite 116 → 127. Ten review candidates were rejected as non-improvements; two confirmed-but-low-value ones were deliberately skipped. The work was Chimera-attributed per PR.