tools(sentry): org-wide volume report + measure on submitted, not accepted (#172)
* tools(sentry): add org-wide volume report and fix the metric we measure on The AGE-105 gate is a measured number, so it needs a repeatable query. It also needed a correction: `accepted` is the wrong headline. The org is over its error quota, so Sentry rejects nearly everything and `accepted` reads ~0 for every project - a blown org and a fixed one look identical on that column. The demand metric is `submitted` = accepted + rate_limited. scripts/sentry-volume-report.mjs takes named --window ranges and prints per-project submitted / accepted / rate_limited / client_discard plus the per-hour and projected per-month rate, so before/after comparisons run the exact same query instead of being re-derived by hand each time. Records the pre-rollout baseline in docs/analytics.md: opencode-mobile at 4.71/h (3,441/mo), 87% of the org's post-box-bot demand, from two windows that agree to within 0.2%. Co-Authored-By: Paperclip <noreply@paperclip.ing> * test(sentry): pin the noise gate against 90d of real production events The gate's unit tests prove it behaves as specified. Nothing proved the spec was aimed at the right targets. Replaying the actual 90d census of the opencode-mobile Sentry project (648 events, 11 issues) through the gate's own precedence shows 96.9% hard-dropped as transport noise and every observed crash class (OOM, ANR, IllegalStateException) still allowlisted -> ~87 events/month against a 1,500/month target. Also records two findings from measuring the org directly: * The error quota resets on the 4th. The 5,000-event month opened 2026-08-04 and was spent by 08-08; the org has accepted zero errors since. 2026-09-04 is the date the gate has to hold by, and it is why 'submitted' is the metric. * Server-side levers are unavailable on this plan. A per-key rate limit PUT returns HTTP 200 and silently discards the value (verified for three window sizes), custom inbound filters are absent, spike protection 403s. The client gate is the only control that exists, so its coverage is the whole margin. Refs AGE-105 Co-Authored-By: Paperclip <noreply@paperclip.ing> * tools(sentry): split client_discard by reason so gate drops aren't confused with quota backoff Raw client_discard cannot show whether the noise gate works. Today 100% of opencode-mobile's client_discard is ratelimit_backoff -- the SDK backing off a 429 because the ORG is over quota -- which rises when things get WORSE. Gate drops land in a different reason: @sentry/core records before_send when beforeSend returns null. - stats_v2 now groups by reason as well as project/outcome - the before_send vs ratelimit_backoff split always prints; --by-reason adds the full per-project reason table - before_send > 0 is install-share-independent, so it proves the gate is live on real devices days before a monthly rate can bend - documents that release-level segmentation is impossible while over quota: rate_limited events are never stored, so release tags stop (last value 0.4.12, 2026-08-08). Version share comes from Play, not Sentry. * ci(sentry): block a Play release whose bundle lost the noise gate The AGE-105 quota fix is entirely client-side (every server-side lever on this plan is dead), so the gate being *in the shipped binary* is the whole safety margin. That is also the one thing Sentry cannot tell us: while the org is over quota nothing is stored, release tags stop dead at 0.4.12, and a release:0.4.14 query returns empty in a way that reads like success. Grep the Hermes bundle inside the AAB instead, before the Play upload step: the gate's reason codes, the transport drop-list regex, the noise.dropped_since_last tag only applyNoiseGate() writes, and a baked-in DSN (a release built without EXPO_PUBLIC_SENTRY_DSN makes Sentry a silent no-op). Verified to discriminate on real artifacts - the v0.4.14 build now on Play production passes, pre-gate v0.4.13 fails all six markers. Also records the rejected alternative: persisting gate state across cold starts pays off only under ~94 active devices (2,633 session envelopes/7d vs a 6h cooldown), and the install base is above that. --------- Co-authored-by: engineer <engineer@macbookpro.lan> Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Den
12
.github/workflows/publish-play-store.yml
vendored
12
.github/workflows/publish-play-store.yml
vendored
@@ -125,6 +125,18 @@ jobs:
|
|||||||
RELEASE_KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
|
RELEASE_KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
|
||||||
run: ./gradlew bundleRelease
|
run: ./gradlew bundleRelease
|
||||||
|
|
||||||
|
- name: Verify the Sentry noise gate is in the artifact
|
||||||
|
# AGE-105: the org error quota is defended ONLY by the client-side gate
|
||||||
|
# (every server-side lever on this Sentry plan was checked and is dead:
|
||||||
|
# per-key rate limit silently no-ops with a 200, custom inbound filters
|
||||||
|
# absent, spike protection 403). If a build ships without the gate — or
|
||||||
|
# without a DSN, which makes Sentry a silent no-op — the org goes back
|
||||||
|
# over quota, and while it is over quota Sentry stores nothing, so the
|
||||||
|
# regression is invisible in Sentry itself until the monthly reset.
|
||||||
|
# Grep the shipped Hermes bundle instead. Verified to discriminate:
|
||||||
|
# v0.4.14 passes, pre-gate v0.4.13 fails.
|
||||||
|
run: node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab
|
||||||
|
|
||||||
- name: Upload AAB artifact
|
- name: Upload AAB artifact
|
||||||
uses: actions/upload-artifact@v7
|
uses: actions/upload-artifact@v7
|
||||||
with:
|
with:
|
||||||
|
|||||||
@@ -115,6 +115,158 @@ Rules are pure and unit-tested in `src/lib/sentry-noise.test.ts` (18 tests, incl
|
|||||||
the observed 1,126-event hour → 5 delivered events). Widening the drop-list is a deliberate
|
the observed 1,126-event hour → 5 delivered events). Widening the drop-list is a deliberate
|
||||||
act: add a test asserting the new pattern, and never add anything that could mask a crash.
|
act: add a test asserting the new pattern, and never add anything that could mask a crash.
|
||||||
|
|
||||||
|
### Measuring whether it worked
|
||||||
|
|
||||||
|
```sh
|
||||||
|
SENTRY_AUTH_TOKEN=… node scripts/sentry-volume-report.mjs --by-reason \
|
||||||
|
--org vibetechnologies \
|
||||||
|
--window "before=2026-08-14T07:00:00Z..2026-08-14T14:00:00Z" \
|
||||||
|
--window "after=2026-08-17T00:00:00Z..now"
|
||||||
|
```
|
||||||
|
|
||||||
|
Read **`submitted` = `accepted` + `rate_limited`**, never `accepted` alone. The org is
|
||||||
|
currently over its error quota, so Sentry rejects essentially everything and `accepted`
|
||||||
|
reads ~0 for *every* project — a blown org and a fixed one look identical on that column.
|
||||||
|
`submitted` is the demand the clients actually put on the wire, which is what the
|
||||||
|
3,500/month gate is really about.
|
||||||
|
|
||||||
|
**And do not read raw `client_discard` as "the gate is working" either — that is the same
|
||||||
|
mistake one column over.** Split it by reason (`--by-reason`, and the split line prints
|
||||||
|
unconditionally):
|
||||||
|
|
||||||
|
| `client_discard` reason | what it means |
|
||||||
|
|---|---|
|
||||||
|
| `before_send` | **our noise gate dropped the event.** Recorded by `@sentry/core` `baseclient.js` whenever `beforeSend` returns `null`. The only proof the gate is live on real devices. |
|
||||||
|
| `ratelimit_backoff` | the SDK is in 429 backoff because the **org** is over quota. A symptom of the overage; it goes UP when things get worse. |
|
||||||
|
| `event_processor`, `network_error` | neither of the above. |
|
||||||
|
|
||||||
|
On 2026-08-14, 100% of `opencode-mobile`'s `client_discard` was `ratelimit_backoff` and
|
||||||
|
`before_send` was 0 — i.e. the pre-rollout `client_discard` number was entirely quota
|
||||||
|
damage, not filtering. So the healthy shape is precisely: `submitted` falls **and**
|
||||||
|
`client_discard/before_send` rises from zero.
|
||||||
|
|
||||||
|
`before_send > 0` is also the **earliest** available evidence, because it does not depend on
|
||||||
|
what share of the install base has updated: one device on v0.4.14 hitting one filtered error
|
||||||
|
produces it. Check it before waiting days for the monthly rate to bend.
|
||||||
|
|
||||||
|
**Do not try to segment the after-number by app release.** While the org is over quota,
|
||||||
|
rate-limited events are never stored, so the project's `release`/`dist` tag values and issue
|
||||||
|
list stop dead (last value: `opencode-mobile@0.4.12`, 2026-08-08) even though clients keep
|
||||||
|
submitting. Version share comes from Play (`scripts/play-version-share.mjs`), not Sentry.
|
||||||
|
|
||||||
|
Pre-rollout baseline for the v0.4.14 comparison (measured 2026-08-14 14:00 UTC, before
|
||||||
|
production rollout at 14:22 UTC), two windows agreeing to within 0.2%:
|
||||||
|
|
||||||
|
| Window | `opencode-mobile` submitted | → /month | Org submitted → /month |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 7h, post-box-bot-fix (08-14 07:00–14:00Z) | 33 (4.71/h) | 3,441 | 3,963 |
|
||||||
|
| 7d trailing (08-07–08-14) | 793 (4.72/h) | 3,446 | 25,450 (box-bot pre-fix) |
|
||||||
|
|
||||||
|
Mobile was 87% of the org's post-box-bot demand. Target: under ~1,500/month, which puts the
|
||||||
|
org under the 3,500/month gate.
|
||||||
|
|
||||||
|
### The quota resets on the 4th — that is the real deadline
|
||||||
|
|
||||||
|
Org-wide daily `accepted` shows a hard billing boundary:
|
||||||
|
|
||||||
|
| Date | org `accepted` | org `rate_limited` | cumulative `accepted` |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 08-03 | 2 | 427 | 24 |
|
||||||
|
| **08-04** | **837** | 10 | 861 |
|
||||||
|
| 08-07 | 1,820 | 0 | 3,812 |
|
||||||
|
| **08-08** | 1,574 | 816 | **5,386** |
|
||||||
|
| 08-09 → 08-14 | 0 | 155–672/day | 5,387 |
|
||||||
|
|
||||||
|
The period reset on **2026-08-04**, the 5,000-error month was spent in **4.5 days**, and the
|
||||||
|
org has been receiving *zero* error data since **2026-08-08**. Next reset: **2026-09-04**.
|
||||||
|
Two consequences: (1) no `accepted`-based measurement is possible before then, which is why
|
||||||
|
`submitted` is the metric; (2) 09-04 is the date the gate actually has to hold by.
|
||||||
|
|
||||||
|
Who spent it, over the 30d to 08-14:
|
||||||
|
|
||||||
|
| project | accepted | rate_limited | submitted |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `openclaw-box-bot` | 4,401 (82%) | 11,741 | 16,142 |
|
||||||
|
| `vibe-api-gateway` | 254 | 3,321 | 3,575 |
|
||||||
|
| `opencode-mobile` | 664 (12%) | 2,148 | 2,812 |
|
||||||
|
| `openclaw-ci` | 68 | 333 | 401 |
|
||||||
|
|
||||||
|
`opencode-mobile` did not blow the quota — `openclaw-box-bot` did (fixed by AGE-55). But with
|
||||||
|
box-bot at 0, mobile is now the dominant remaining demand.
|
||||||
|
|
||||||
|
### Server-side levers do not exist on this plan
|
||||||
|
|
||||||
|
Checked directly against the API on 2026-08-14, so nobody re-litigates it:
|
||||||
|
|
||||||
|
| Lever | Result |
|
||||||
|
|---|---|
|
||||||
|
| Per-key rate limit (`PUT /projects/{org}/{proj}/keys/{id}/`) | **Silent no-op.** Returns HTTP 200 and drops the field; a follow-up GET always reads `rateLimit: null`. Reproduced with `window` = 60, 3600 and 86400. |
|
||||||
|
| Custom inbound filters (error message / release) | Not present. Only the five generic browser filters exist. |
|
||||||
|
| Spike protection (`/organizations/{org}/spike-protections/`) | HTTP 403. |
|
||||||
|
|
||||||
|
Org `features` is `[]`. **The client-side gate is the only control that exists**, so its
|
||||||
|
coverage is the entire safety margin — which is why `sentry-noise-production.test.ts` pins
|
||||||
|
that coverage against real production data.
|
||||||
|
|
||||||
|
### Gate coverage against 90d of real events
|
||||||
|
|
||||||
|
Every issue in the project over the 90d to 2026-08-14 (648 events), replayed through the
|
||||||
|
gate's own precedence (allowlist first, then drop-list) by
|
||||||
|
`src/lib/sentry-noise-production.test.ts`:
|
||||||
|
|
||||||
|
| Outcome | events | share |
|
||||||
|
|---|---|---|
|
||||||
|
| hard-dropped as transport noise | 628 | 96.9% |
|
||||||
|
| always-send crash classes (OOM, ANR, `IllegalStateException`) | 13 | 2.0% |
|
||||||
|
| deduped / rate-capped (the 401 storm) | 7 | 1.1% |
|
||||||
|
|
||||||
|
Upper bound of surviving volume: 2,750/month × 3.1% ≈ **87/month**, ~17× under the
|
||||||
|
1,500/month target, and that ignores dedup and the hourly cap, which only push it lower.
|
||||||
|
Crash classes still come through — the test fails if any of them stops being allowlisted,
|
||||||
|
because hitting the number by silencing real crashes is a failure, not a win.
|
||||||
|
|
||||||
|
### The one piece of evidence available on release day: check the artifact
|
||||||
|
|
||||||
|
Everything above is worthless if a build ships without the gate. That failure is
|
||||||
|
*invisible in Sentry*: while the org is over quota nothing is stored, so release tags stop
|
||||||
|
updating (opencode-mobile's stop at `0.4.12` / 2026-08-08 while clients keep submitting
|
||||||
|
~4.7/h) and a `release:0.4.14` query returns an empty result that reads exactly like "no
|
||||||
|
errors from the new build".
|
||||||
|
|
||||||
|
The binary is checkable the same day. Hermes bytecode keeps string literals, so
|
||||||
|
`scripts/verify-release-bundle.mjs` greps `base/assets/index.android.bundle` inside the AAB
|
||||||
|
for the gate's own reason codes, the transport drop-list regex, and the
|
||||||
|
`noise.dropped_since_last` tag that only `applyNoiseGate()` writes — plus the baked-in DSN,
|
||||||
|
because a release built without `EXPO_PUBLIC_SENTRY_DSN` makes `Sentry.init()` a silent
|
||||||
|
no-op. It runs in `publish-play-store.yml` **before** the Play upload step, so a gateless
|
||||||
|
build cannot reach users.
|
||||||
|
|
||||||
|
It discriminates — this is not a self-confirming assertion:
|
||||||
|
|
||||||
|
| Artifact | Result |
|
||||||
|
|---|---|
|
||||||
|
| v0.4.14 AAB, versionCode 151 (the build now on Play production, run `31807432647`) | **passes**, all six markers, DSN = project `4511436292292608` |
|
||||||
|
| v0.4.13 AAB (run `31786473735`, pre-gate) | **fails**, all six markers absent |
|
||||||
|
|
||||||
|
### Rejected: persisting gate state across launches
|
||||||
|
|
||||||
|
The rate caps (`maxPerHour`, `maxNewPerHour`) and the 6h per-fingerprint cooldown live in
|
||||||
|
process memory, so every cold start resets them. That looks like a hole worth plugging
|
||||||
|
with `AsyncStorage`; the session data says it is not.
|
||||||
|
|
||||||
|
Sentry session envelopes for `opencode-mobile`, 7d to 2026-08-14: **2,633** (267–541/day),
|
||||||
|
which is the install base's app-start rate. Persisting only pays off if a device launches
|
||||||
|
the app *more often than the cooldown expires* — i.e. if `376 launches/day ÷ devices > 4`,
|
||||||
|
so only below **~94 active devices**. One issue alone (`connect timeout`) has 104 distinct
|
||||||
|
users over 90d, so the install base is above that line and the two rates are within
|
||||||
|
rounding of each other. Persisting would add native storage I/O on the crash path to buy
|
||||||
|
nothing measurable. Revisit only if the app-start rate rises well above ~4/device/day.
|
||||||
|
|
||||||
|
(Those session envelopes are 100% `client_discard`, 96% `network_error` — sent at cold
|
||||||
|
start and at process teardown, when the transport often can't complete. Sessions are not
|
||||||
|
billed, so this costs no quota, but it does mean release health is not a usable signal for
|
||||||
|
this app either.)
|
||||||
|
|
||||||
## Disclosure surfaces (must stay in sync)
|
## Disclosure surfaces (must stay in sync)
|
||||||
|
|
||||||
| Surface | File |
|
| Surface | File |
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
"android": "expo run:android",
|
"android": "expo run:android",
|
||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
"check:versions": "node scripts/check-version-parity.mjs",
|
"check:versions": "node scripts/check-version-parity.mjs",
|
||||||
"test": "node --test 'src/**/*.test.ts'"
|
"test": "node --test 'src/**/*.test.ts' 'scripts/*.test.mjs'"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@expo/vector-icons": "^15.0.3",
|
"@expo/vector-icons": "^15.0.3",
|
||||||
|
|||||||
228
scripts/sentry-volume-report.mjs
Normal file
228
scripts/sentry-volume-report.mjs
Normal file
@@ -0,0 +1,228 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// Org-wide Sentry error-volume report.
|
||||||
|
//
|
||||||
|
// Answers one question: how many error events per month is this org actually
|
||||||
|
// consuming, per project, per outcome — and how did that change across a
|
||||||
|
// deploy boundary?
|
||||||
|
//
|
||||||
|
// Why it exists: the "Sentry event budget" work (see docs/analytics.md) is
|
||||||
|
// gated on a *measured* rate, not on "the filter is merged". Every heartbeat
|
||||||
|
// that re-checks the number should run the same query, or the before/after
|
||||||
|
// comparison is not a comparison.
|
||||||
|
//
|
||||||
|
// Usage:
|
||||||
|
// SENTRY_AUTH_TOKEN=... node scripts/sentry-volume-report.mjs \
|
||||||
|
// --by-reason --org vibetechnologies \
|
||||||
|
// --window "pre=2026-08-13T14:00:00Z..2026-08-14T06:00:00Z" \
|
||||||
|
// --window "post=2026-08-14T14:22:00Z..now"
|
||||||
|
//
|
||||||
|
// Notes on reading the output:
|
||||||
|
// * The headline metric is `submitted` = accepted + rate_limited: every event
|
||||||
|
// the client actually put on the wire, i.e. real demand against quota.
|
||||||
|
// Do NOT headline `accepted`. Once the org is over quota, Sentry rejects
|
||||||
|
// everything and `accepted` collapses to ~0 for every project — which
|
||||||
|
// makes a broken org look identical to a fixed one.
|
||||||
|
// * `rate_limited` is demand that arrived after the quota was gone. It is
|
||||||
|
// evidence of a problem, not of a fix.
|
||||||
|
// * `client_discard` is NOT a gate metric on its own. Split it by reason
|
||||||
|
// (`--by-reason`, on by default in the summary line):
|
||||||
|
// - `before_send` -> our noise gate dropped the event. THIS is the
|
||||||
|
// only column that proves the gate is running on
|
||||||
|
// real devices, and it is independent of install
|
||||||
|
// -base share, so it shows up long before the
|
||||||
|
// monthly rate bends.
|
||||||
|
// - `ratelimit_backoff` -> the SDK is in 429 backoff because the ORG is
|
||||||
|
// over quota. Pure symptom of the overage. It
|
||||||
|
// rises when things get WORSE. Reading raw
|
||||||
|
// `client_discard` as "the gate is working" is
|
||||||
|
// the same class of error as reading `accepted`.
|
||||||
|
// - `event_processor` / `network_error` -> neither of the above.
|
||||||
|
// * Per-release attribution is impossible while the org is over quota:
|
||||||
|
// rate_limited events are never stored, so `release`/`dist` tag values (and
|
||||||
|
// issues) simply stop. Do not try to segment the after-number by app
|
||||||
|
// version from Sentry; use outcome+reason, and Play for version share.
|
||||||
|
// * A window shorter than ~24h cannot certify a monthly rate; it can only
|
||||||
|
// rank sources. Diurnal load is real.
|
||||||
|
|
||||||
|
const API = "https://sentry.io/api/0"
|
||||||
|
const MONTH_HOURS = 730
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const out = { org: process.env.SENTRY_ORG ?? "vibetechnologies", windows: [] }
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i]
|
||||||
|
if (a === "--org") out.org = argv[++i]
|
||||||
|
else if (a === "--window") out.windows.push(argv[++i])
|
||||||
|
else if (a === "--json") out.json = true
|
||||||
|
else if (a === "--by-reason") out.byReason = true
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseWindow(spec) {
|
||||||
|
const eq = spec.indexOf("=")
|
||||||
|
if (eq < 0) throw new Error(`bad --window ${spec} (want name=START..END)`)
|
||||||
|
const name = spec.slice(0, eq)
|
||||||
|
const [rawStart, rawEnd] = spec.slice(eq + 1).split("..")
|
||||||
|
if (!rawStart || !rawEnd) throw new Error(`bad --window ${spec} (want name=START..END)`)
|
||||||
|
const at = (v) => (v === "now" ? new Date() : new Date(v))
|
||||||
|
const start = at(rawStart)
|
||||||
|
const end = at(rawEnd)
|
||||||
|
if (Number.isNaN(start.getTime()) || Number.isNaN(end.getTime())) {
|
||||||
|
throw new Error(`bad --window ${spec} (unparseable date)`)
|
||||||
|
}
|
||||||
|
if (end <= start) throw new Error(`bad --window ${spec} (end <= start)`)
|
||||||
|
return { name, start, end, hours: (end - start) / 3_600_000 }
|
||||||
|
}
|
||||||
|
|
||||||
|
async function sentry(path, token) {
|
||||||
|
const res = await fetch(`${API}${path}`, { headers: { Authorization: `Bearer ${token}` } })
|
||||||
|
const body = await res.json()
|
||||||
|
if (!res.ok) throw new Error(`${path} -> ${res.status} ${JSON.stringify(body)}`)
|
||||||
|
return body
|
||||||
|
}
|
||||||
|
|
||||||
|
async function projectSlugs(org, token) {
|
||||||
|
const projects = await sentry(`/organizations/${org}/projects/`, token)
|
||||||
|
return new Map(projects.map((p) => [String(p.id), p.slug]))
|
||||||
|
}
|
||||||
|
|
||||||
|
async function windowStats(org, token, win) {
|
||||||
|
const qs = new URLSearchParams({
|
||||||
|
field: "sum(quantity)",
|
||||||
|
category: "error",
|
||||||
|
start: win.start.toISOString().replace(/\.\d+Z$/, "Z"),
|
||||||
|
end: win.end.toISOString().replace(/\.\d+Z$/, "Z"),
|
||||||
|
project: "-1",
|
||||||
|
})
|
||||||
|
qs.append("groupBy", "project")
|
||||||
|
qs.append("groupBy", "outcome")
|
||||||
|
qs.append("groupBy", "reason")
|
||||||
|
const data = await sentry(`/organizations/${org}/stats_v2/?${qs}`, token)
|
||||||
|
const rows = new Map()
|
||||||
|
for (const g of data.groups ?? []) {
|
||||||
|
const key = String(g.by.project)
|
||||||
|
if (!rows.has(key)) rows.set(key, { outcomes: {}, reasons: {} })
|
||||||
|
const row = rows.get(key)
|
||||||
|
const qty = g.totals["sum(quantity)"] ?? 0
|
||||||
|
if (!qty) continue
|
||||||
|
row.outcomes[g.by.outcome] = (row.outcomes[g.by.outcome] ?? 0) + qty
|
||||||
|
const reason = g.by.reason ?? "none"
|
||||||
|
row.reasons[`${g.by.outcome}/${reason}`] = (row.reasons[`${g.by.outcome}/${reason}`] ?? 0) + qty
|
||||||
|
}
|
||||||
|
return rows
|
||||||
|
}
|
||||||
|
|
||||||
|
function fmt(n) {
|
||||||
|
return n.toLocaleString("en-US", { maximumFractionDigits: 0 })
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const args = parseArgs(process.argv.slice(2))
|
||||||
|
const token = process.env.SENTRY_AUTH_TOKEN
|
||||||
|
if (!token) {
|
||||||
|
console.error("SENTRY_AUTH_TOKEN is not set. This script is read-only; a scoped read token is enough.")
|
||||||
|
process.exit(2)
|
||||||
|
}
|
||||||
|
if (args.windows.length === 0) {
|
||||||
|
// Default: last 7 days, one window. Enough to certify a monthly rate.
|
||||||
|
const end = new Date()
|
||||||
|
const start = new Date(end.getTime() - 7 * 86_400_000)
|
||||||
|
args.windows.push(`7d=${start.toISOString()}..${end.toISOString()}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const windows = args.windows.map(parseWindow)
|
||||||
|
const slugs = await projectSlugs(args.org, token)
|
||||||
|
const results = []
|
||||||
|
for (const win of windows) {
|
||||||
|
results.push({ win, rows: await windowStats(args.org, token, win) })
|
||||||
|
}
|
||||||
|
|
||||||
|
const report = { org: args.org, generatedAt: new Date().toISOString(), windows: [] }
|
||||||
|
for (const { win, rows } of results) {
|
||||||
|
const projects = []
|
||||||
|
let orgSubmitted = 0
|
||||||
|
for (const [id, { outcomes, reasons }] of rows) {
|
||||||
|
const accepted = outcomes.accepted ?? 0
|
||||||
|
const rateLimited = outcomes.rate_limited ?? 0
|
||||||
|
const submitted = accepted + rateLimited
|
||||||
|
orgSubmitted += submitted
|
||||||
|
projects.push({
|
||||||
|
project: slugs.get(id) ?? id,
|
||||||
|
accepted,
|
||||||
|
rateLimited,
|
||||||
|
submitted,
|
||||||
|
clientDiscard: outcomes.client_discard ?? 0,
|
||||||
|
// The gate. Everything else in client_discard is not us.
|
||||||
|
gateDropped: reasons["client_discard/before_send"] ?? 0,
|
||||||
|
backoffDropped: reasons["client_discard/ratelimit_backoff"] ?? 0,
|
||||||
|
reasons,
|
||||||
|
filtered: outcomes.filtered ?? 0,
|
||||||
|
submittedPerHour: submitted / win.hours,
|
||||||
|
submittedPerMonth: (submitted / win.hours) * MONTH_HOURS,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
projects.sort((a, b) => b.submitted - a.submitted)
|
||||||
|
report.windows.push({
|
||||||
|
name: win.name,
|
||||||
|
start: win.start.toISOString(),
|
||||||
|
end: win.end.toISOString(),
|
||||||
|
hours: Number(win.hours.toFixed(2)),
|
||||||
|
projects,
|
||||||
|
orgSubmitted,
|
||||||
|
orgSubmittedPerMonth: (orgSubmitted / win.hours) * MONTH_HOURS,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if (args.json) {
|
||||||
|
console.log(JSON.stringify(report, null, 2))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const w of report.windows) {
|
||||||
|
console.log(`\n== ${w.name} ${w.start} -> ${w.end} (${w.hours}h)`)
|
||||||
|
console.log(
|
||||||
|
`${"project".padEnd(24)}${"submitted".padStart(11)}${"accepted".padStart(10)}${"rate_lim".padStart(10)}${"cli_disc".padStart(10)}${"sub/h".padStart(9)}${"sub/mo".padStart(10)}`,
|
||||||
|
)
|
||||||
|
for (const p of w.projects) {
|
||||||
|
console.log(
|
||||||
|
`${p.project.padEnd(24)}${fmt(p.submitted).padStart(11)}${fmt(p.accepted).padStart(10)}${fmt(p.rateLimited).padStart(10)}${fmt(p.clientDiscard).padStart(10)}${p.submittedPerHour.toFixed(2).padStart(9)}${fmt(p.submittedPerMonth).padStart(10)}`,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
console.log(
|
||||||
|
`${"ORG TOTAL (submitted)".padEnd(24)}${fmt(w.orgSubmitted).padStart(11)}${"".padStart(30)}${(w.orgSubmitted / w.hours).toFixed(2).padStart(9)}${fmt(w.orgSubmittedPerMonth).padStart(10)}`,
|
||||||
|
)
|
||||||
|
|
||||||
|
// Always print the gate-liveness split, because raw `client_discard` is
|
||||||
|
// ambiguous: `ratelimit_backoff` (org over quota, a symptom) looks exactly
|
||||||
|
// like `before_send` (our gate, the fix) unless you split them.
|
||||||
|
const gate = w.projects.reduce((n, p) => n + p.gateDropped, 0)
|
||||||
|
const backoff = w.projects.reduce((n, p) => n + p.backoffDropped, 0)
|
||||||
|
console.log(
|
||||||
|
`client_discard split: before_send (our gate) ${fmt(gate)} | ratelimit_backoff (org over quota) ${fmt(backoff)}`,
|
||||||
|
)
|
||||||
|
if (gate === 0) {
|
||||||
|
console.log(" before_send == 0 -> no device is running the noise gate yet in this window.")
|
||||||
|
}
|
||||||
|
|
||||||
|
if (args.byReason) {
|
||||||
|
for (const p of w.projects) {
|
||||||
|
const entries = Object.entries(p.reasons).sort((a, b) => b[1] - a[1])
|
||||||
|
if (entries.length === 0) continue
|
||||||
|
console.log(` ${p.project}`)
|
||||||
|
for (const [k, v] of entries) console.log(` ${fmt(v).padStart(8)} ${k}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log(
|
||||||
|
"\nGate: org submitted (accepted + rate_limited) < 3,500/month." +
|
||||||
|
"\nWindows under ~24h rank sources but do not certify a rate." +
|
||||||
|
"\nGate liveness: client_discard/before_send > 0 for opencode-mobile." +
|
||||||
|
"\nDo NOT segment by release: over-quota events are never stored, so release tags stop.",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((err) => {
|
||||||
|
console.error(err.message)
|
||||||
|
process.exit(1)
|
||||||
|
})
|
||||||
137
scripts/verify-release-bundle.mjs
Normal file
137
scripts/verify-release-bundle.mjs
Normal file
@@ -0,0 +1,137 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// Verify that a release AAB actually ships the Sentry noise gate (AGE-105).
|
||||||
|
//
|
||||||
|
// Why this exists: the entire quota fix is client-side. Sentry's server-side
|
||||||
|
// levers were checked and are all dead on this plan (per-key rate limit returns
|
||||||
|
// HTTP 200 and silently drops the field; custom inbound filters absent; spike
|
||||||
|
// protection 403). So if a build ever ships without the gate — a Metro entry
|
||||||
|
// change, a refactor that stops importing sentry-noise, a release built without
|
||||||
|
// EXPO_PUBLIC_SENTRY_DSN — the org quietly goes back over quota and nobody
|
||||||
|
// finds out until the monthly reset.
|
||||||
|
//
|
||||||
|
// It also can't be caught by watching Sentry: while the org is over quota,
|
||||||
|
// events are rejected at ingest and never stored, so release tags stop
|
||||||
|
// updating (opencode-mobile's stop at 0.4.12 / 2026-08-08 while clients keep
|
||||||
|
// submitting ~4.7/h). A `release:0.4.14` query returns EMPTY, which reads like
|
||||||
|
// "no errors from the new build" but actually means "no data at all".
|
||||||
|
//
|
||||||
|
// The binary itself is the only same-day evidence. Hermes bytecode keeps string
|
||||||
|
// literals, so the gate's own reason strings and the drop-list regex are
|
||||||
|
// greppable in base/assets/index.android.bundle.
|
||||||
|
//
|
||||||
|
// Usage:
|
||||||
|
// node scripts/verify-release-bundle.mjs android/app/build/outputs/bundle/release/app-release.aab
|
||||||
|
//
|
||||||
|
// Optional env:
|
||||||
|
// EXPO_PUBLIC_SENTRY_DSN when set, the DSN baked into the bundle must point
|
||||||
|
// at the same project id (never printed).
|
||||||
|
//
|
||||||
|
// Read-only, dependency-free (uses the `unzip` CLI, present on the runner).
|
||||||
|
|
||||||
|
import { execFileSync } from "node:child_process"
|
||||||
|
import { existsSync } from "node:fs"
|
||||||
|
|
||||||
|
/** Literals that only exist because the noise gate is in the bundle graph.
|
||||||
|
* Each is unique to our code, not the Sentry SDK. */
|
||||||
|
export const GATE_MARKERS = [
|
||||||
|
// NoiseGate reason codes (src/lib/sentry-noise.ts)
|
||||||
|
"transport-noise",
|
||||||
|
"new-fingerprint-cap",
|
||||||
|
"hourly-cap",
|
||||||
|
"always-send",
|
||||||
|
// the transport drop-list regex source — the single biggest volume cut
|
||||||
|
"connect (?:timeout",
|
||||||
|
// applyNoiseGate() in src/lib/sentry.ts, i.e. the gate is wired to beforeSend
|
||||||
|
"noise.dropped_since_last",
|
||||||
|
]
|
||||||
|
|
||||||
|
const DSN_RE = /https:\/\/[0-9a-f]{16,64}@[a-z0-9.-]*ingest[a-z0-9.-]*\/(\d+)/g
|
||||||
|
|
||||||
|
/** Pure check over the raw JS/Hermes bundle bytes. Returns a list of problems;
|
||||||
|
* empty means the artifact is good. Never returns the DSN itself. */
|
||||||
|
export function checkBundle(bundle, opts = {}) {
|
||||||
|
const text = Buffer.isBuffer(bundle) ? bundle.toString("latin1") : String(bundle)
|
||||||
|
const problems = []
|
||||||
|
|
||||||
|
const missing = GATE_MARKERS.filter((marker) => !text.includes(marker))
|
||||||
|
if (missing.length) {
|
||||||
|
problems.push(`noise gate missing from bundle — absent markers: ${missing.join(", ")}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const projectIds = [...text.matchAll(DSN_RE)].map((m) => m[1])
|
||||||
|
if (projectIds.length === 0) {
|
||||||
|
// No DSN => Sentry.init() is a no-op => the gate never runs and no telemetry
|
||||||
|
// arrives at all. A release build must never be in this state.
|
||||||
|
problems.push("no Sentry DSN baked into the bundle — telemetry would be a silent no-op")
|
||||||
|
} else if (opts.expectedProjectId && !projectIds.includes(String(opts.expectedProjectId))) {
|
||||||
|
problems.push(
|
||||||
|
`bundled DSN points at project ${projectIds.join("/")}, expected ${opts.expectedProjectId}`,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return problems
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Extract the project id from a DSN without leaking the key. */
|
||||||
|
export function dsnProjectId(dsn) {
|
||||||
|
if (!dsn) return null
|
||||||
|
const m = /\/(\d+)\/?$/.exec(String(dsn).trim())
|
||||||
|
return m ? m[1] : null
|
||||||
|
}
|
||||||
|
|
||||||
|
export function checkAppConfig(configJson, expectedVersion) {
|
||||||
|
if (!expectedVersion) return []
|
||||||
|
let version
|
||||||
|
try {
|
||||||
|
version = JSON.parse(configJson)?.version
|
||||||
|
} catch {
|
||||||
|
return ["base/assets/app.config is not valid JSON"]
|
||||||
|
}
|
||||||
|
return version === expectedVersion ? [] : [`bundle app.config version ${version}, expected ${expectedVersion}`]
|
||||||
|
}
|
||||||
|
|
||||||
|
function unzipEntry(archive, entry) {
|
||||||
|
return execFileSync("unzip", ["-p", archive, entry], { maxBuffer: 256 * 1024 * 1024 })
|
||||||
|
}
|
||||||
|
|
||||||
|
function main() {
|
||||||
|
const aab = process.argv[2]
|
||||||
|
if (!aab || !existsSync(aab)) {
|
||||||
|
console.error("usage: node scripts/verify-release-bundle.mjs <path-to.aab|.apk>")
|
||||||
|
process.exit(2)
|
||||||
|
}
|
||||||
|
|
||||||
|
let bundle
|
||||||
|
try {
|
||||||
|
bundle = unzipEntry(aab, "base/assets/index.android.bundle")
|
||||||
|
} catch {
|
||||||
|
console.error(`FAIL ${aab}: no base/assets/index.android.bundle inside the archive`)
|
||||||
|
process.exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
const problems = checkBundle(bundle, { expectedProjectId: dsnProjectId(process.env.EXPO_PUBLIC_SENTRY_DSN) })
|
||||||
|
|
||||||
|
let version = null
|
||||||
|
try {
|
||||||
|
const config = unzipEntry(aab, "base/assets/app.config").toString("utf8")
|
||||||
|
version = JSON.parse(config)?.version ?? null
|
||||||
|
} catch {
|
||||||
|
problems.push("could not read base/assets/app.config")
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`artifact: ${aab}`)
|
||||||
|
console.log(`bundle: ${(bundle.length / 1024 / 1024).toFixed(2)} MiB, app version ${version ?? "unknown"}`)
|
||||||
|
for (const marker of GATE_MARKERS) {
|
||||||
|
const ok = bundle.toString("latin1").includes(marker)
|
||||||
|
console.log(` ${ok ? "ok " : "MISS"} ${marker}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
if (problems.length) {
|
||||||
|
console.error("\nFAIL — this build must not go to Play:")
|
||||||
|
for (const p of problems) console.error(` - ${p}`)
|
||||||
|
process.exit(1)
|
||||||
|
}
|
||||||
|
console.log("\nOK — Sentry noise gate is present and pointed at the expected project.")
|
||||||
|
}
|
||||||
|
|
||||||
|
if (import.meta.url === `file://${process.argv[1]}`) main()
|
||||||
64
scripts/verify-release-bundle.test.mjs
Normal file
64
scripts/verify-release-bundle.test.mjs
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
import assert from "node:assert/strict"
|
||||||
|
import test from "node:test"
|
||||||
|
|
||||||
|
import { GATE_MARKERS, checkAppConfig, checkBundle, dsnProjectId } from "./verify-release-bundle.mjs"
|
||||||
|
|
||||||
|
const DSN = "https://4f21a8b3c4d5e6f708192a3b4c5d6e7f@o4510132673511424.ingest.us.sentry.io/4511436292292608"
|
||||||
|
const PROJECT = "4511436292292608"
|
||||||
|
|
||||||
|
function goodBundle(extra = "") {
|
||||||
|
return Buffer.from(`\u0000\u0001hermes${GATE_MARKERS.join("\u0000")}\u0000${DSN}${extra}`, "latin1")
|
||||||
|
}
|
||||||
|
|
||||||
|
test("a bundle with every gate marker and the right DSN passes", () => {
|
||||||
|
assert.deepEqual(checkBundle(goodBundle(), { expectedProjectId: PROJECT }), [])
|
||||||
|
})
|
||||||
|
|
||||||
|
test("a bundle built without the gate fails and names the missing markers", () => {
|
||||||
|
const stripped = Buffer.from(`hermes${DSN}`, "latin1")
|
||||||
|
const problems = checkBundle(stripped, { expectedProjectId: PROJECT })
|
||||||
|
assert.equal(problems.length, 1)
|
||||||
|
for (const marker of GATE_MARKERS) assert.match(problems[0], new RegExp(marker.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")))
|
||||||
|
})
|
||||||
|
|
||||||
|
test("dropping a single marker is enough to fail — a partial gate is not a gate", () => {
|
||||||
|
const partial = Buffer.from(`hermes${GATE_MARKERS.slice(1).join("|")}${DSN}`, "latin1")
|
||||||
|
const problems = checkBundle(partial, { expectedProjectId: PROJECT })
|
||||||
|
assert.equal(problems.length, 1)
|
||||||
|
assert.match(problems[0], /transport-noise/)
|
||||||
|
})
|
||||||
|
|
||||||
|
test("a release built without EXPO_PUBLIC_SENTRY_DSN fails — Sentry would be a silent no-op", () => {
|
||||||
|
const noDsn = Buffer.from(`hermes${GATE_MARKERS.join("|")}`, "latin1")
|
||||||
|
const problems = checkBundle(noDsn, { expectedProjectId: PROJECT })
|
||||||
|
assert.deepEqual(problems, ["no Sentry DSN baked into the bundle — telemetry would be a silent no-op"])
|
||||||
|
})
|
||||||
|
|
||||||
|
test("a DSN for the wrong project fails, and the failure never prints the key", () => {
|
||||||
|
const wrong = goodBundle().toString("latin1").replace(PROJECT, "9999999999")
|
||||||
|
const problems = checkBundle(Buffer.from(wrong, "latin1"), { expectedProjectId: PROJECT })
|
||||||
|
assert.equal(problems.length, 1)
|
||||||
|
assert.match(problems[0], /expected 4511436292292608/)
|
||||||
|
assert.ok(!problems[0].includes("4f21a8b3c4d5e6f708192a3b4c5d6e7f"), "problem text must not leak the DSN key")
|
||||||
|
})
|
||||||
|
|
||||||
|
test("without an expected project id the DSN only has to exist", () => {
|
||||||
|
assert.deepEqual(checkBundle(goodBundle(), {}), [])
|
||||||
|
})
|
||||||
|
|
||||||
|
test("dsnProjectId extracts the trailing id and tolerates junk", () => {
|
||||||
|
assert.equal(dsnProjectId(DSN), PROJECT)
|
||||||
|
assert.equal(dsnProjectId(`${DSN}/`), PROJECT)
|
||||||
|
assert.equal(dsnProjectId(""), null)
|
||||||
|
assert.equal(dsnProjectId(undefined), null)
|
||||||
|
assert.equal(dsnProjectId("not-a-dsn"), null)
|
||||||
|
})
|
||||||
|
|
||||||
|
test("app.config version mismatch is reported, match is silent", () => {
|
||||||
|
assert.deepEqual(checkAppConfig(JSON.stringify({ version: "0.4.14" }), "0.4.14"), [])
|
||||||
|
assert.deepEqual(checkAppConfig(JSON.stringify({ version: "0.4.13" }), "0.4.14"), [
|
||||||
|
"bundle app.config version 0.4.13, expected 0.4.14",
|
||||||
|
])
|
||||||
|
assert.deepEqual(checkAppConfig("{", "0.4.14"), ["base/assets/app.config is not valid JSON"])
|
||||||
|
assert.deepEqual(checkAppConfig("{", null), [])
|
||||||
|
})
|
||||||
168
src/lib/sentry-noise-production.test.ts
Normal file
168
src/lib/sentry-noise-production.test.ts
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
// Replay of REAL production Sentry data through the AGE-105 noise gate.
|
||||||
|
//
|
||||||
|
// `sentry-noise.test.ts` proves the gate behaves as specified. This file proves
|
||||||
|
// the specification was aimed at the right targets — that the patterns match the
|
||||||
|
// event signatures `opencode-mobile` actually emits, at the volumes it actually
|
||||||
|
// emits them, and therefore that the gate lands under the org quota gate.
|
||||||
|
//
|
||||||
|
// Why freeze a census instead of querying Sentry in a test: the numbers below
|
||||||
|
// are the *justification* for the pattern lists. If someone later edits
|
||||||
|
// TRANSPORT_NOISE_PATTERNS or ALWAYS_SEND_PATTERNS and the projected volume
|
||||||
|
// leaves budget — or a genuine crash class stops being allowlisted — that is a
|
||||||
|
// regression, and it should fail here rather than on next month's Sentry bill.
|
||||||
|
// A live query would move under our feet and could never fail.
|
||||||
|
//
|
||||||
|
// Census: every issue in the `opencode-mobile` Sentry project over the 90 days
|
||||||
|
// ending 2026-08-14, sorted by event count, taken from
|
||||||
|
// GET /organizations/vibetechnologies/issues/?statsPeriod=90d&sort=freq.
|
||||||
|
// Reproduce with `node scripts/sentry-volume-report.mjs` + the issues endpoint.
|
||||||
|
//
|
||||||
|
// Volume context measured the same day (see docs/analytics.md):
|
||||||
|
// * `opencode-mobile` submitted = 2,812 events / 30d (~2,750/month)
|
||||||
|
// * org error quota 5,000/month, self-imposed gate 3,500/month (AGE-71)
|
||||||
|
// * AGE-105 target for this project: under 1,500/month
|
||||||
|
//
|
||||||
|
// NOTE: server-side levers were tested and are NOT available on this Sentry
|
||||||
|
// plan — per-key rate limits accept a PUT with HTTP 200 and silently discard
|
||||||
|
// the value, custom inbound filters are absent, spike protection returns 403.
|
||||||
|
// This client-side gate is the only control that exists, so its coverage is the
|
||||||
|
// whole safety margin.
|
||||||
|
|
||||||
|
import { test } from "node:test"
|
||||||
|
import assert from "node:assert/strict"
|
||||||
|
import { eventText, isAlwaysSend, isTransportNoise } from "./sentry-noise.ts"
|
||||||
|
|
||||||
|
type Observed = {
|
||||||
|
/** Sentry short id, so a failure is traceable to the real issue. */
|
||||||
|
id: string
|
||||||
|
/** exception type as Sentry grouped it */
|
||||||
|
type: string
|
||||||
|
/** exception value, verbatim (truncated exactly as Sentry stores it) */
|
||||||
|
value: string
|
||||||
|
/** events in the 90d window */
|
||||||
|
count: number
|
||||||
|
/** what the gate must decide, and why it is the correct decision */
|
||||||
|
expect: "drop" | "always-send" | "capped"
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Frozen 90d production census (648 events across 11 issues). */
|
||||||
|
const PRODUCTION_90D: Observed[] = [
|
||||||
|
// --- client-side network conditions: already shown to the user as connection
|
||||||
|
// UI and already trended in PostHog as connection_failed{error_class}. ---
|
||||||
|
{ id: "OPENCODE-MOBILE-3", type: "Error", value: "connect timeout", count: 462, expect: "drop" },
|
||||||
|
{ id: "OPENCODE-MOBILE-4", type: "Error", value: "connect server-unreachable", count: 157, expect: "drop" },
|
||||||
|
{ id: "OPENCODE-MOBILE-B", type: "Error", value: "Network request failed", count: 6, expect: "drop" },
|
||||||
|
{ id: "OPENCODE-MOBILE-7", type: "Error", value: "Request timed out after 30000ms", count: 3, expect: "drop" },
|
||||||
|
|
||||||
|
// --- genuine crash classes: must survive whatever the quota says. ---
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-A",
|
||||||
|
type: "Error",
|
||||||
|
value:
|
||||||
|
"Call to function 'ExponentImagePicker.launchImageLibraryAsync' has been rejected.\n→ Caused by: java.lang.IllegalStateException: Attempting to launch an unregistered ActivityResultLauncher with contract expo.modules.imagepicker.contracts.ImageLibraryC",
|
||||||
|
count: 5,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-5",
|
||||||
|
type: "OutOfMemoryError",
|
||||||
|
value: "Failed to allocate a 126790640 byte allocation with 12451840 free bytes",
|
||||||
|
count: 4,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-6",
|
||||||
|
type: "OutOfMemoryError",
|
||||||
|
value: "Failed to allocate a 8208 byte allocation with 1261584 free bytes",
|
||||||
|
count: 1,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-8",
|
||||||
|
type: "ApplicationNotResponding",
|
||||||
|
value: "ANR",
|
||||||
|
count: 1,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-9",
|
||||||
|
type: "IllegalStateException",
|
||||||
|
value: "The specified child already has a parent. You must call removeView() on the child's parent first.",
|
||||||
|
count: 1,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "OPENCODE-MOBILE-2",
|
||||||
|
type: "IllegalStateException",
|
||||||
|
value: "The specified child already has a parent. You must call removeView() on the child's parent first.",
|
||||||
|
count: 1,
|
||||||
|
expect: "always-send",
|
||||||
|
},
|
||||||
|
|
||||||
|
// --- reportable, but a storm of it must collapse: AGE-107 traced 498 of
|
||||||
|
// these to one user re-tapping Connect with a wrong password. Dedup + rate
|
||||||
|
// cap handle it; a hard drop would hide real server-side 401 regressions. ---
|
||||||
|
{ id: "OPENCODE-MOBILE-1", type: "Error", value: "API Error: 401 - ", count: 7, expect: "capped" },
|
||||||
|
]
|
||||||
|
|
||||||
|
const asEvent = (o: Observed) => ({
|
||||||
|
exception: { values: [{ type: o.type, value: o.value, mechanism: { handled: true } }] },
|
||||||
|
})
|
||||||
|
|
||||||
|
function classify(o: Observed): Observed["expect"] {
|
||||||
|
const text = eventText(asEvent(o))
|
||||||
|
// Mirrors NoiseGate.admit() precedence: allowlist wins over the drop-list.
|
||||||
|
if (isAlwaysSend(text)) return "always-send"
|
||||||
|
if (isTransportNoise(text)) return "drop"
|
||||||
|
return "capped"
|
||||||
|
}
|
||||||
|
|
||||||
|
test("production replay: every observed issue is classified as intended", () => {
|
||||||
|
for (const o of PRODUCTION_90D) {
|
||||||
|
assert.equal(
|
||||||
|
classify(o),
|
||||||
|
o.expect,
|
||||||
|
`${o.id} (${o.count} events, ${o.type}: ${o.value.slice(0, 60)}) should be ${o.expect}`,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test("production replay: no genuine crash class is ever filtered", () => {
|
||||||
|
// The failure mode this guards: hitting the volume target by silencing real
|
||||||
|
// crashes. A zero here would make the quota win meaningless.
|
||||||
|
const crashes = PRODUCTION_90D.filter((o) => o.expect === "always-send")
|
||||||
|
assert.ok(crashes.length >= 6, "census should still contain the observed crash classes")
|
||||||
|
for (const o of crashes) {
|
||||||
|
assert.equal(isAlwaysSend(eventText(asEvent(o))), true, `${o.id} must be allowlisted`)
|
||||||
|
assert.equal(classify(o), "always-send", `${o.id} must not be reachable by the drop-list`)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test("production replay: gate drops >=95% of observed volume", () => {
|
||||||
|
let dropped = 0
|
||||||
|
let total = 0
|
||||||
|
for (const o of PRODUCTION_90D) {
|
||||||
|
total += o.count
|
||||||
|
if (classify(o) === "drop") dropped += o.count
|
||||||
|
}
|
||||||
|
assert.equal(total, 648, "census total changed — re-derive the projections below")
|
||||||
|
const share = dropped / total
|
||||||
|
assert.ok(share >= 0.95, `expected >=95% of volume dropped, got ${(share * 100).toFixed(1)}%`)
|
||||||
|
})
|
||||||
|
|
||||||
|
test("production replay: projected monthly volume clears the AGE-105 target", () => {
|
||||||
|
const SUBMITTED_PER_MONTH = 2750 // measured 2026-08-14, pre-v0.4.14-uptake
|
||||||
|
const TARGET_PER_MONTH = 1500 // AGE-105 "done means"
|
||||||
|
let survivingShare = 0
|
||||||
|
let total = 0
|
||||||
|
for (const o of PRODUCTION_90D) {
|
||||||
|
total += o.count
|
||||||
|
if (classify(o) !== "drop") survivingShare += o.count
|
||||||
|
}
|
||||||
|
// Upper bound: assumes dedup + the hourly cap never fire, which they will.
|
||||||
|
const projected = SUBMITTED_PER_MONTH * (survivingShare / total)
|
||||||
|
assert.ok(
|
||||||
|
projected < TARGET_PER_MONTH,
|
||||||
|
`projected ${Math.round(projected)}/month must stay under ${TARGET_PER_MONTH}/month`,
|
||||||
|
)
|
||||||
|
})
|
||||||
Reference in New Issue
Block a user