tools(sentry): anchor the measurement windows on the gate's rollout instant (#178)

A window that spans the 2026-08-14 14:22Z production rollout contains devices
that could not possibly have run the gate. Its rate is neither a baseline nor a
result, and it prints identically to both. This was not hypothetical: a
`post=08-14T07:00Z..now` window (84% of it pre-rollout) was run against this
script and reported opencode-mobile *rising* to 4.44/h.

`noise-gate-report.mjs` shipped with the same defect built into its default:
`post = now-7d..now` straddles the rollout on every run before 08-21, diluting
the after-rate toward baseline - biased toward grading the gate as ineffective
on exactly the dates the ticket schedules its reads (08-17, 08-21).

- sentry-volume-report: `--since-rollout` reads the instant from the release
  history table in docs/playstore.md (production versionCode >= 150, earliest
  such release, so a later v0.4.15 does not restart the window) and splits
  there. Every window is labelled [pre]/[post]/[mixed]; mixed prints how much
  of it predates the gate, a young post window prints its uptake age, and an
  unparseable table reports "unknown" rather than assuming post.
- noise-gate-report: defaults post to the rollout instant, returns UNGRADED for
  a mixed/unknown post window, and pins the baseline to the documented
  post-box-bot-fix window instead of a 7d lookback that dragged ~22k/mo of
  already-fixed box-bot volume into the org outlook (it read "MISSES by 18,612"
  for a dead reason; now 628/mo, clears).
- before_send == 0 is now reported as expected in a pre/mixed/young window and
  as a failure only after 24h+ of gated production.

Re-probed every server-side lever with a WRITE-scoped token so none of the
answers is a permissions artifact, and corrected the record in docs/analytics.md:
per-key rate limit returns 200 and silently drops the field; error-message
filters return 400 "You do not have that feature enabled" (a plan gate, not
absence - it is the one lever that would reach never-updating installs); spike
protection is not 403-unavailable, it is already enabled everywhere and simply
does not fire on sustained baseline volume.

Co-authored-by: engineer <engineer@macbookpro.lan>
This commit is contained in:
Den
2026-08-14 10:54:43 -07:00
committed by GitHub
parent c43ec27a8c
commit 4d64700b1b
7 changed files with 462 additions and 41 deletions

View File

@@ -133,13 +133,16 @@ jobs:
run: ./gradlew bundleRelease
- name: Verify the Sentry noise gate is in the artifact
# AGE-105: the org error quota is defended ONLY by the client-side gate
# (every server-side lever on this Sentry plan was checked and is dead:
# per-key rate limit silently no-ops with a 200, custom inbound filters
# absent, spike protection 403). If a build ships without the gate — or
# without a DSN, which makes Sentry a silent no-op — the org goes back
# over quota, and while it is over quota Sentry stores nothing, so the
# regression is invisible in Sentry itself until the monthly reset.
# AGE-105: the org error quota is defended ONLY by the client-side gate.
# Every server-side lever on this Sentry plan was probed and is dead
# (docs/analytics.md): per-key rate limit answers 200 and silently drops
# the field, custom error-message filters answer 400 "You do not have
# that feature enabled", and spike protection is already on everywhere
# but only catches spikes, not this steady baseline. If a build ships
# without the gate — or without a DSN, which makes Sentry a silent
# no-op — the org goes back over quota, and while it is over quota
# Sentry stores nothing, so the regression is invisible in Sentry itself
# until the monthly reset.
# Grep the shipped Hermes bundle instead. Verified to discriminate:
# v0.4.14 passes, pre-gate v0.4.13 fails.
run: node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab

View File

@@ -83,6 +83,6 @@ jobs:
echo ''
echo '### Raw org volume'
echo '```'
node scripts/sentry-volume-report.mjs --by-reason || true
node scripts/sentry-volume-report.mjs --by-reason --since-rollout || true
echo '```'
} >> "$GITHUB_STEP_SUMMARY"