octoscope
Your GitHub profile as a live terminal dashboard
Start new thread

octoscope 0.27.0 — the scan stops describing and starts noticing

by
•

octoscope 0.27.0 — the integrity scan stops describing a repo and starts noticing things about it.

Before this release the supply-chain scan could only talk in the present tense: here is what auto-executes right now, here is a commit tip that looks forged. Useful if you happen to scan while the implant is sitting there. Useless for the two questions you actually have when a worm is circulating — did something change, and what could whoever got in reach?

Now: every scan records a fingerprint of what auto-executes in the repo, and the next one tells you what moved. A file that auto-executes appeared. An existing one's contents changed. A branch tip that used to be signed no longer is. It survives renames and re-obfuscation, because a variant still has to appear — and appearing is the thing being watched.

octoscope

It also maps your capability footprint: workflow permissions and triggers, self-hosted runners, write deploy keys, webhooks delivering somewhere other than GitHub.

A tip that is really a design note: holding power is not a finding. My own release workflow asks for contents:write and reads two secrets, and it is correct — it fires on a tag push, so only someone who can already push tags can reach it. What scores is power reachable from untrusted input: a pull_request_target workflow holding your secrets, or a self-hosted runner an outsider's pull request can actually target. Scoring power by itself would flag a large share of GitHub and teach everyone to ignore the whole thing.

The detail I enjoyed getting right: the first scan of a repo says so out loud. There is no baseline yet, so there is nothing to compare — and staying quiet there would be indistinguishable from "nothing changed". A security tool that cannot tell you the difference between "I looked and it is fine" and "I had nothing to look at" is worse than one that admits it. The same reasoning made the checks needing admin scope fail open but still name themselves: "deploy keys (the token lacks the scope this needs)".

brew upgrade gfazioli/tap/octoscope

So: how would you find out today that something which auto-executes changed in one of your repos? I keep asking people this and the honest answer is usually "I wouldn't". Curious whether that matches your experience.

170 views

Add a comment

Replies

Best

I appreciate that you avoid flagging every permission as dangerous. too many scanners create alert fatigue. How did you decide what should actually affect the score?

 that was basically the whole design argument, and the answer came from my own repo.

octoscope's release workflow asks for contents:write and reads two secrets. completely normal — it fires on a tag push, so the only people who can reach it are people who could already push tags. if that scored, a big chunk of github lights up and everyone learns to ignore the axis inside a week.

so the rule became: capability isn't a finding, capability reachable from untrusted input is. same file triggered by pull_request_target → now it's a privilege-escalation gadget. and a self-hosted runner only counts if a fork-triggered job actually targets self-hosted, not just because both happen to exist in the same repo — otherwise the report claims outsider code can run on your hardware when it can't.

on top of that there's a hard ceiling: no single axis can reach the "suspicious" tier alone, it needs a second independent signal to agree. that one's enforced by a test rather than by good intentions — the first weighting i wrote broke it (two findings from the same axis summed to 6 against a threshold of 5) and a reviewer caught it. i lowered the weights instead of raising the threshold, which felt like the honest direction

 I really like that principle reachability matters more than raw permissions. Requiring independent signals before calling something suspicious also feels like a solid way to reduce false positives. It's interesting that a reviewer caught the weighting issue did that change how you approach designing new detection rules now?

 yeah — the rule that came out of it is that a test pinning a ceiling on a sum has to build the maximal case, because one term tells you nothing about the total.

and I know it stuck because it just caught the same thing again, in that exact test. it built one workflow, which scores 4 — the ceiling value itself — so it couldn't tell "the clamp held" from "these weights are small anyway". I disabled the clamp and it still passed. now it enumerates every contributor and asserts the ceiling is saturated. that's #109, open right now.

so the honest answer is that the reviewer changed what I check, not just what I write.

I never really thought about monitoring changes to autov executing files over time. that feels much more practical than taking isolated snapshots.

 yeah, that was exactly the shift. a snapshot answers "what's here right now" — and you can stare at a perfectly clean-looking snapshot of a repo that got hit last tuesday and learn nothing from it.

the part i like about the diff is that it's the one check that doesn't care what the payload looks like. rename the dropper, obfuscate it differently, pick a new path — it still has to appear, and appearing is the thing being watched. filename lists age badly. "something that auto-executes showed up since last time" doesn't.

curious have you already tested this against repositories that were compromised in the past? I would be interested to know how well it catches those scenarios.

 partly, yeah. the per-repo content scan was validated against the actual (since cleaned) victims of the reference worm plus unaffected controls — everything scored clean after remediation, which was the correct answer rather than a miss, since the payload was already gone by then...

the detector itself is built on the invariant of that attack class from Ionut-Cristian Florescu's writeups — he's the maintainer whose own repos got hijacked — rather than a filename list, which is why renamed variants still trip it.

but i should be straight about the delta axis specifically: it can't be validated that way by construction. you'd need a fingerprint from before the compromise, and a cleaned repo only ever gives you the "after". best i have is synthetic — doctoring a baseline and confirming the appearance and signature-regression paths fire and actually escalate the verdict.

if you know of a repo with history from both sides of an incident, i'd genuinely like to run it.

 Thanks for the detailed explanation. I like that you're upfront about what can and can't be validated that builds more confidence than overclaiming. Hopefully a real before-and-after incident history surfaces at some point; it'd be a fascinating benchmark for the delta detection.

 Here's the grim part: it probably won't surface on its own, because the correct cleanup destroys it. The remediation octoscope hands you is reset to the clean parent, force-push, then ask GitHub Support to garbage-collect — a revert would leave the payload retrievable at the old commit. Right answer, but it removes the "during" a benchmark needs.

So the only real before-and-after comes from someone who happened to be scanning already. That's an adoption problem, not a data one — strange thing to want, but I do.

I'm impressed by the shift from describing to noticing. For me, that sounds like scans becoming genuinely useful instead of just informative.

 thanks — "informative vs genuinely useful" is a better framing than the one i had, might steal it.

the thing that tipped it for me was actually the boring case: if there's no baseline yet, the report says so out loud. sounds trivial, but silence there is indistinguishable from "nothing changed" — and a tool that can't tell you the difference between "i looked and it's fine" and "i had nothing to look at" isn't really informative either.

I've always felt raw scan results need context. I'm curious what kinds of patterns Octoscope 0.27.0 now notices automatically that previous versions would have overlooked.

 sure — concretely, and split by why they were missed before:

couldn't be seen at all before, because there was no memory of a previous scan:

- a file that auto-executes appeared on a branch that already existed

- a known one's contents changed (same path, different blob)

- a branch tip that used to carry a real author signature no longer does

could have been seen, just wasn't being looked at:

- a fork-triggered workflow (pull_request_target / workflow_run / issue_comment) holding the repo's secrets or write scopes

- a self-hosted runner that one of those workflows can actually target

- a cross-repo push burst — but only ever as corroboration for a repo that already scored on something else

the first group is the one that needed the fingerprint, and it's the group i'd bet on long term: it survives renames and re-obfuscation for free

Out of curiosity, whats been the most surprising thing you have discovered while testing Octoscope against public repositories?

 the one that actually changed the code: github silently signs some commits for you.

the naive rule seems obvious — if a repo signs its commits and one branch tip doesn't, that's worth a look.. except a web-UI edit or a squash-merge gets a GitHub signature automatically, so a repo where the maintainer has never once touched GPG still looks like "a repo that signs". and then every ordinary unsigned feature braanch reads as a signing regression.

I hit it on one of the actual (since cleaned) victim repos i was validating against, which is the funniest possible place to find a false positive. fix was that only genuine author signatures count toward the baseline — signed AND not-signed-by-github...

second one, less of a gotcha but it shaped the entire scoring model: how much of a perfectly normal repo auto-executes. the catalog has 21 patterns; a boring go repo trips 2, but a real js-heavy account lights up with postinstall scripts, editor task files, agent hook configs — and essentially all of it is legitimate.

That's the reason no single axis can push the verdict up on its own. if it could, the tool would be noise inside a week and everyone would learn to ignore it

 That's a really interesting edge case. I probably would've made the same assumption about GitHub signatures. I also like the philosophy behind the scoring model avoiding alert fatigue is just as important as catching real issues. Sounds like validating against real repositories taught you far more than synthetic test cases ever could.

 Half true, and the split matters. Real repos taught me what normal looks like — that's a false-positive lesson, and the signature thing is exactly it. What they cant teach is false negatives, which in a security tool is the expensive direction to be wrong in.

Best example came from neither: a unit test asserting "no single axis reaches suspicious alone" sat green while the invariant was broken. It built one workflow; two findings from the same axis summed past the threshold. A reviewer caught that, not the suite.

Also worth saying the cleaned victim repos couldn't validate the delta axis at all — nothing pre-compromise to diff against. Newest axis, least checkable against the real thing.

I like your philosophy that holding power isnt a finding. Context matters far more than raw permissions, and many tools seem to miss that.

 thanks! worth naming the cost though — it's precision over recall, on purpose. a scanner that flags every write permission will catch things mine deliberately won't, and if your threat model is "audit who can do what across the org", that one's the better tool

mine is pointed at "did something get in", where a report nobody reads is worth exactly zero

The release notes made me rethink how I look at GitHub security. i usually focus on secrets, but workflow reachability seems just as important.

 That's the frame i'd push too, and it took me embarrassingly long to get there.

A secret is only worth what can reach it. Ten secrets in a repo with no fork-triggered workflow is a smaller problem than one secret in a repo with a pull_request_target job that checks out PR code. Rotating is the reflex because it's the thing you can actually do in 30 seconds — but if the path stays open, the new secret is reachable by exactly the same route.

it's the same shape as the remediation mistake in the incident this whole detector was built from: people revoked the leaked token and moved on, while the OAuth grant stayed authorised. new token, same door. reachability is that idea one layer up.

Wouldn't either, honestly. We lean on branch protection and required reviews, but that only catches things a human actually reads in a diff. A compromised dependency shipping a postinstall script wouldn't show up as a code change at all, it'd show up as a package-lock diff nobody reads line by line. Never thought about tracking auto-execute surface as its own signal separate from the code diff, that's a real gap in what we actually watch.

 Yeah, that's the split that made me build it as its own axis.

Presence alone is weight 0 though — every repo has a postinstall and a tasks.json, so scoring that just trains you to ignore it. What scores is the delta: fingerprint the auto-exec paths, diff against the last scan, flag what changed. Doesn't care about filenames or contents, just notices that something which runs by itself moved.

tbh your example is the one case it doesn't cover — I don't read lockfiles at all. Threat model is a payload in the source of a repo you own, not a bad package arriving from the registry. Dependency posture is the open direction (#58), parked on token scope more than effort: Dependabot alerts want security_events, way past what octoscope asks for. 👍 it if you'd use it

 Makes sense once you say it, the two threats need completely different signals, a payload already in your own source doesn't produce a registry event to catch. Resets what I'd actually want, sounds like I need both, something watching what's already in the repo plus something else on the dependency side. Will keep an eye on #58 if it lands.

 yeah, both is the right read — and the dependency half already has decent tools, dependabot and npm audit do that better than I would.

worth saying the two stages aren't as separate as they look though: shai-hulud style worms re-push themselves into the repos you own once they land, so the registry sees the way in and the delta sees the propagation. that second stage is the one nobody's watching.

if you've got a shape in mind for #58, dropping it as a comment there is worth more than the upvote.

Moving from present tense to history is the whole difference between a scanner and a witness. A point in time scan can only ever tell you the state when you happened to look, which is exactly the wrong sampling for anything that arrives and leaves. The implant that mattered was there on a Tuesday. What I would want out of this: not just that something changed, but that it changed and then changed back. Nothing legitimate reverts itself quietly, so a value that returns to its original after a window is a much stronger signal than either state on its own, and it is invisible to anyone comparing only against the current tip. Does 0.27 keep enough history to catch a revert, or does it compare against last scan?

 compares against the last scan, and that's the unflattering answer — one fingerpint per repo, overwritten every time. so A → B → A observed across three scans just reads as two ordinary changes, and the second one prints an OID identical to what the first scan stored without knowing it. the same round trip happening entirely between two scans, your Tuesday implant, produces nothing at all 😅

cheap to fix though, and that's down to your framing: the value we store is already a content hash, so a bit-identical revert reproduces the exact OID. detection is a lookup, not a new mechanism. The real question is how far back to keep it — a baseline already stops scoring past 30 days, since by then most of the diff is my own commits.

worth conceding too, the scan is on-demand rather than a cron, so the sampling window is whenever someone presses the key. sharpens your point rather than softens it.

anyway, really glad you wrote this one. "nothing legitimate reverts itself quietly" is imo the sentence that turns it into a signal — and no, it isn't tracked as an issue yet. it should be.

 On how far back to keep it: I would stop treating retention and scoring as the same knob, because they have very different costs.

Scoring past 30 days is genuinely useless, for the reason you give. Storing past 30 days costs one hash per repo per scan, which is nothing. So keep the OID history unbounded and score only inside the window. A revert older than the window then produces a note rather than a finding: this exact content was here before, on this date. Not a security claim, just a fact the reader can weigh, and it costs almost nothing to be able to say it.

The reason I would push for unbounded retention specifically is that patience is the entire point of the attack. Anything willing to revert itself quietly is willing to wait, so a fixed lookback is a published dwell time. Thirty days is a number someone can simply exceed.

On the scan being on-demand rather than cron: agreed it sharpens rather than softens. Though it does mean the fingerprint history records when someone was paying attention, not what happened. Two scans a year apart look identical in the store to two scans a minute apart. Stamping the gap next to the finding would let a reader tell "unchanged across a year" from "unchanged across a coffee break", and those are very different sentences.

 ok, went and checked the code rather than answering from memory, and it splits three ways: one piece of what you're proposing already exists, one genuinely doesn't, and one thing I have to push back on.

the scoring half is already there. the delta axis has a 30-day cutoff, and past it the finding is still emitted — just with weight 0, and the reason string it prints ends with "so this is reported without affecting the verdict". the comment sitting next to the constant says the deltas stay reported with the age stated, they just stop moving the verdict. that is your note-rather-than-a-finding, near enough word for word, and I hadn't connected it to what you were asking for until I went looking 🙂

retention is the part that genuinely isn't there, and it's the ask. one fingerprint per repo, overwritten every scan, so A → B → A survives as two ordinary changes and the first A is simply gone. you're right about the cost too — it's a hash per repo per scan, which is nothing against what it buys.

on the store not knowing time, I'll push back gently: captured_at is persisted per repo, so two scans a year apart don't look identical in there. what's true is narrower and still your point — the age is only printed when the baseline is stale or its capture time is unreadable. inside the window it prints nothing at all, so a finding from a one-minute gap and one from a 29-day gap read exactly the same. stamping it unconditionally is the fix and it's a formatting change, not a mechanism.

"a fixed lookback is a published dwell time" is the line I'll be quoting back at myself, tbh. thirty days got picked because past it most of the diff is my own commits and scoring that noise would train me to ignore the axis — which is an argument about scoring, and I'd quietly let it decide retention as well. two questions, one number. that's the actual contribution here.

one trade-off worth adding, and it sharpens yours rather than answering it: with an on-demand scan the note can only honestly read "this exact content was observed here on this date". unbounded retention makes the note possible; it doesn't make the history continuous. someone patient enough to outlast a lookback is patient enough to move between two scans that happen to sit six months apart — so the note has to carry the same gap stamp you're asking for on the finding, or it quietly overclaims.

filed as two rather than one, which is your doing: for unbounded retention with scoring kept inside the window, and for stamping the gap on every delta. both credit you, and 126 carries your dwell-time argument as the reason it's unbounded rather than just a bigger number. thanks — turning "keep more history" into two knobs with different costs is what made it buildable instead of just agreeable.