2cc284ecbe
* tools(sentry): add org-wide volume report and fix the metric we measure on The AGE-105 gate is a measured number, so it needs a repeatable query. It also needed a correction: `accepted` is the wrong headline. The org is over its error quota, so Sentry rejects nearly everything and `accepted` reads ~0 for every project - a blown org and a fixed one look identical on that column. The demand metric is `submitted` = accepted + rate_limited. scripts/sentry-volume-report.mjs takes named --window ranges and prints per-project submitted / accepted / rate_limited / client_discard plus the per-hour and projected per-month rate, so before/after comparisons run the exact same query instead of being re-derived by hand each time. Records the pre-rollout baseline in docs/analytics.md: opencode-mobile at 4.71/h (3,441/mo), 87% of the org's post-box-bot demand, from two windows that agree to within 0.2%. Co-Authored-By: Paperclip <noreply@paperclip.ing> * test(sentry): pin the noise gate against 90d of real production events The gate's unit tests prove it behaves as specified. Nothing proved the spec was aimed at the right targets. Replaying the actual 90d census of the opencode-mobile Sentry project (648 events, 11 issues) through the gate's own precedence shows 96.9% hard-dropped as transport noise and every observed crash class (OOM, ANR, IllegalStateException) still allowlisted -> ~87 events/month against a 1,500/month target. Also records two findings from measuring the org directly: * The error quota resets on the 4th. The 5,000-event month opened 2026-08-04 and was spent by 08-08; the org has accepted zero errors since. 2026-09-04 is the date the gate has to hold by, and it is why 'submitted' is the metric. * Server-side levers are unavailable on this plan. A per-key rate limit PUT returns HTTP 200 and silently discards the value (verified for three window sizes), custom inbound filters are absent, spike protection 403s. The client gate is the only control that exists, so its coverage is the whole margin. Refs AGE-105 Co-Authored-By: Paperclip <noreply@paperclip.ing> * tools(sentry): split client_discard by reason so gate drops aren't confused with quota backoff Raw client_discard cannot show whether the noise gate works. Today 100% of opencode-mobile's client_discard is ratelimit_backoff -- the SDK backing off a 429 because the ORG is over quota -- which rises when things get WORSE. Gate drops land in a different reason: @sentry/core records before_send when beforeSend returns null. - stats_v2 now groups by reason as well as project/outcome - the before_send vs ratelimit_backoff split always prints; --by-reason adds the full per-project reason table - before_send > 0 is install-share-independent, so it proves the gate is live on real devices days before a monthly rate can bend - documents that release-level segmentation is impossible while over quota: rate_limited events are never stored, so release tags stop (last value 0.4.12, 2026-08-08). Version share comes from Play, not Sentry. * ci(sentry): block a Play release whose bundle lost the noise gate The AGE-105 quota fix is entirely client-side (every server-side lever on this plan is dead), so the gate being *in the shipped binary* is the whole safety margin. That is also the one thing Sentry cannot tell us: while the org is over quota nothing is stored, release tags stop dead at 0.4.12, and a release:0.4.14 query returns empty in a way that reads like success. Grep the Hermes bundle inside the AAB instead, before the Play upload step: the gate's reason codes, the transport drop-list regex, the noise.dropped_since_last tag only applyNoiseGate() writes, and a baked-in DSN (a release built without EXPO_PUBLIC_SENTRY_DSN makes Sentry a silent no-op). Verified to discriminate on real artifacts - the v0.4.14 build now on Play production passes, pre-gate v0.4.13 fails all six markers. Also records the rejected alternative: persisting gate state across cold starts pays off only under ~94 active devices (2,633 session envelopes/7d vs a 6h cooldown), and the install base is above that. --------- Co-authored-by: engineer <engineer@macbookpro.lan> Co-authored-by: Paperclip <noreply@paperclip.ing>
18 KiB
Activation Analytics — Design Record
Design record for the PostHog-based activation-funnel analytics added to OpenCode Mobile,
and how it is disclosed and consent-gated. Companion to docs/playstore.md (Data safety)
and distribution/privacy-policy.md (user-facing policy). GitHub issue: #63.
Note: the same consent flag also gates a third, separate data flow not covered by this doc: delivery of user-shared diagnostic reports to our self-hosted Chatwoot support inbox (
src/lib/chatwoot.ts,src/lib/diagnostics.ts, issue #85/#88). That flow is triggered manually ("Share Report"), not automatic like Sentry/PostHog. It is disclosed alongside Sentry and PostHog in every surface in the table below; seedistribution/privacy-policy.md§3b for the full description.
Goal
Answer one product question: do new users successfully connect to their opencode server and reach first value (message sent → response received)? Nothing else is tracked.
SDK and destination
| Item | Value |
|---|---|
| SDK | posthog-react-native, self-instantiated (no PostHogProvider, no autocapture) |
| Destination | PostHog EU region — https://eu.i.posthog.com (override: EXPO_PUBLIC_POSTHOG_HOST) |
| API key | EXPO_PUBLIC_POSTHOG_KEY (CI secret; unset ⇒ analytics is a strict no-op) |
| Identity | PostHog's random app-generated anonymous ID only; no identify() calls, no user IDs |
| Code | src/lib/analytics.ts (wrapper), src/lib/analytics-classify.ts (error bucketing), src/lib/demo-analytics.ts (demo-funnel property derivation), src/lib/telemetry.ts (consent gate) |
Event schema
Keep this table in 1:1 sync with AnalyticsEvent in src/lib/analytics.ts and with
section 3a of distribution/privacy-policy.md.
| Event | Fired when | Properties | Call site |
|---|---|---|---|
app_opened |
Once per JS session, as soon as analytics is enabled (cold start with prior consent, or immediately after consent grant) | is_first_open: boolean |
app/_layout.tsx, src/lib/telemetry.ts |
connection_form_submitted |
User taps Connect/Save with a non-empty server URL | mode: "quick" | "advanced" |
app/connection/add.tsx |
connection_attempted |
A real connection test starts (advanced mode: fired on save, no pre-flight check) | source: "onboarding" | "edit_test" |
src/stores/connections.ts, app/connection/add.tsx |
connection_succeeded |
Health check responds OK | source |
src/stores/connections.ts |
connection_failed |
Health check fails | source, error_class |
src/stores/connections.ts |
message_sent |
User sends a prompt to an agent session (excludes slash commands) | — | src/stores/sessions.ts |
response_received |
Agent response finishes streaming (busy → idle), excluding user-aborted runs | — | src/stores/events.ts |
demo_started |
The offline /demo screen mounts (no server, no network) |
— | app/demo.tsx |
demo_step_advanced |
User advances a step in the scripted demo (currently: replies to the demo's permission prompt) | step_index, step_name, reply ("once" | "always" | "reject") |
app/demo.tsx, src/lib/demo-analytics.ts |
demo_completed |
The scripted demo reaches its end (completion or denial message shown) — the key demo activation metric | outcome ("completed" | "denied") |
app/demo.tsx, src/lib/demo-analytics.ts |
demo_exited_to_connect |
User taps "Connect your own server" on the demo's CTA card | reached_completion (boolean) |
app/demo.tsx |
error_class is one of a fixed enum — malformed-url, no-internet, server-unreachable,
unauthorized, tls-error, timeout, unknown (src/lib/analytics-classify.ts). The raw
error string is never sent (it can embed hostnames/IPs/tokens).
PII rule: properties are flat primitives only (AnalyticsProps). Never add server URLs,
hostnames, ports, prompts, message/file content, tokens, or raw error text. Adding any new
event or property requires updating the privacy policy (section 3a) and the consent modal
copy in the same PR.
Consent gating
Single consent flag (opencode_telemetry_consent in expo-secure-store) gates both
Sentry and PostHog — there is no separate analytics toggle. Managed by src/lib/telemetry.ts.
- Off by default. First launch shows
TelemetryConsentModal(discloses crash reports AND usage analytics). No SDK is initialised before a "granted" decision. - Grant:
initSentry()+initAnalytics();app_openedfires (once-per-session guard). - Decline / never asked:
track()is a strict no-op; the PostHog client is never created; nothing is written locally (the first-open flag is only touched post-consent). - Revoke (Settings → Privacy → Crash Reports & Usage Analytics):
- Sentry client closed.
- PostHog: buffered-but-unsent events are DROPPED, not flushed.
ConsentGatedPostHogoverrides the SDKfetch()transport; after revocation every request short-circuits to a synthetic 200, soshutdown()drains the queue with zero bytes leaving the device. SDKoptOut()is persisted first so a re-created client can't capture either.
- Re-grant mid-session:
optIn()clears the persisted opt-out; theapp_openedsession guard prevents double-counting.
Sentry event budget — the noise gate (AGE-105)
Consent decides whether we report; the noise gate in src/lib/sentry-noise.ts decides how
often. It exists because this app became the org's #1 Sentry volume source (~4,500
events/month against a 3,500/month org quota) while ~1,100 of those events were three
non-defects: connect timeout, connect server-unreachable, and one device's
API Error: 401 firing 498 times.
AGE-107 postscript. That 401 storm was traced to a human retry loop, not a client token-refresh loop. In v0.4.4 the connection probe scored any HTTP response as a success, so a 401 was reported to the user as "Health endpoint responded — connection actually works now" while their password was wrong. They re-tapped Connect for two months (Sentry breadcrumbs show a
touchevent before every single capture, at irregular human-paced intervals).requireOkindiagnostics.ts(v0.4.8) stopped the false success;auth-failednow gives it its own actionable message and drop-list entry. The client's automated loops were never at fault —events.tsalready terminates the SSE reconnect loop onApiAuthError(issue #76).
beforeSend applies three layers, cheapest first:
| Layer | Rule | Effect |
|---|---|---|
| Always-send allowlist | OOM / ANR / native / IllegalStateException / NullPointerException / fatal level / unhandled mechanism |
Bypasses every limit below — quota is worthless if it silences real crashes |
| Transport drop-list | connect timeout|server-unreachable|no-internet|malformed-url|auth-failed, Network request failed, Request timed out after, ECONN*/ETIMEDOUT… |
Hard drop. Not sampled: the gate is per-install, so even 1/device/day multiplies by the install base back into thousands/month |
| Dedup + rate cap | per-fingerprint cooldown 6h, ≤6 new fingerprints/h, ≤10 events/h (mirrors the openclaw-box-bot shim, AGE-55) |
Turns a retry loop into one report and caps any future regression |
Nothing is lost by the transport drop: those failures are already shown to the user as
connection UI and already trended, PII-free, as the PostHog connection_failed event with
an error_class property (src/lib/analytics-classify.ts — a 401 lands in unauthorized).
Sentry was paying per event for a graph we already have. connect health-failed and
connect tls-error are deliberately not dropped: a box that answers but is unhealthy, or
a broken certificate, is actionable.
Dropped-event counts are not silent — the number dropped since the last delivered event rides
along as a noise.dropped_since_last tag, so the saving is auditable from Sentry itself.
Rules are pure and unit-tested in src/lib/sentry-noise.test.ts (18 tests, incl. a replay of
the observed 1,126-event hour → 5 delivered events). Widening the drop-list is a deliberate
act: add a test asserting the new pattern, and never add anything that could mask a crash.
Measuring whether it worked
SENTRY_AUTH_TOKEN=… node scripts/sentry-volume-report.mjs --by-reason \
--org vibetechnologies \
--window "before=2026-08-14T07:00:00Z..2026-08-14T14:00:00Z" \
--window "after=2026-08-17T00:00:00Z..now"
Read submitted = accepted + rate_limited, never accepted alone. The org is
currently over its error quota, so Sentry rejects essentially everything and accepted
reads ~0 for every project — a blown org and a fixed one look identical on that column.
submitted is the demand the clients actually put on the wire, which is what the
3,500/month gate is really about.
And do not read raw client_discard as "the gate is working" either — that is the same
mistake one column over. Split it by reason (--by-reason, and the split line prints
unconditionally):
client_discard reason |
what it means |
|---|---|
before_send |
our noise gate dropped the event. Recorded by @sentry/core baseclient.js whenever beforeSend returns null. The only proof the gate is live on real devices. |
ratelimit_backoff |
the SDK is in 429 backoff because the org is over quota. A symptom of the overage; it goes UP when things get worse. |
event_processor, network_error |
neither of the above. |
On 2026-08-14, 100% of opencode-mobile's client_discard was ratelimit_backoff and
before_send was 0 — i.e. the pre-rollout client_discard number was entirely quota
damage, not filtering. So the healthy shape is precisely: submitted falls and
client_discard/before_send rises from zero.
before_send > 0 is also the earliest available evidence, because it does not depend on
what share of the install base has updated: one device on v0.4.14 hitting one filtered error
produces it. Check it before waiting days for the monthly rate to bend.
Do not try to segment the after-number by app release. While the org is over quota,
rate-limited events are never stored, so the project's release/dist tag values and issue
list stop dead (last value: opencode-mobile@0.4.12, 2026-08-08) even though clients keep
submitting. Version share comes from Play (scripts/play-version-share.mjs), not Sentry.
Pre-rollout baseline for the v0.4.14 comparison (measured 2026-08-14 14:00 UTC, before production rollout at 14:22 UTC), two windows agreeing to within 0.2%:
| Window | opencode-mobile submitted |
→ /month | Org submitted → /month |
|---|---|---|---|
| 7h, post-box-bot-fix (08-14 07:00–14:00Z) | 33 (4.71/h) | 3,441 | 3,963 |
| 7d trailing (08-07–08-14) | 793 (4.72/h) | 3,446 | 25,450 (box-bot pre-fix) |
Mobile was 87% of the org's post-box-bot demand. Target: under ~1,500/month, which puts the org under the 3,500/month gate.
The quota resets on the 4th — that is the real deadline
Org-wide daily accepted shows a hard billing boundary:
| Date | org accepted |
org rate_limited |
cumulative accepted |
|---|---|---|---|
| 08-03 | 2 | 427 | 24 |
| 08-04 | 837 | 10 | 861 |
| 08-07 | 1,820 | 0 | 3,812 |
| 08-08 | 1,574 | 816 | 5,386 |
| 08-09 → 08-14 | 0 | 155–672/day | 5,387 |
The period reset on 2026-08-04, the 5,000-error month was spent in 4.5 days, and the
org has been receiving zero error data since 2026-08-08. Next reset: 2026-09-04.
Two consequences: (1) no accepted-based measurement is possible before then, which is why
submitted is the metric; (2) 09-04 is the date the gate actually has to hold by.
Who spent it, over the 30d to 08-14:
| project | accepted | rate_limited | submitted |
|---|---|---|---|
openclaw-box-bot |
4,401 (82%) | 11,741 | 16,142 |
vibe-api-gateway |
254 | 3,321 | 3,575 |
opencode-mobile |
664 (12%) | 2,148 | 2,812 |
openclaw-ci |
68 | 333 | 401 |
opencode-mobile did not blow the quota — openclaw-box-bot did (fixed by AGE-55). But with
box-bot at 0, mobile is now the dominant remaining demand.
Server-side levers do not exist on this plan
Checked directly against the API on 2026-08-14, so nobody re-litigates it:
| Lever | Result |
|---|---|
Per-key rate limit (PUT /projects/{org}/{proj}/keys/{id}/) |
Silent no-op. Returns HTTP 200 and drops the field; a follow-up GET always reads rateLimit: null. Reproduced with window = 60, 3600 and 86400. |
| Custom inbound filters (error message / release) | Not present. Only the five generic browser filters exist. |
Spike protection (/organizations/{org}/spike-protections/) |
HTTP 403. |
Org features is []. The client-side gate is the only control that exists, so its
coverage is the entire safety margin — which is why sentry-noise-production.test.ts pins
that coverage against real production data.
Gate coverage against 90d of real events
Every issue in the project over the 90d to 2026-08-14 (648 events), replayed through the
gate's own precedence (allowlist first, then drop-list) by
src/lib/sentry-noise-production.test.ts:
| Outcome | events | share |
|---|---|---|
| hard-dropped as transport noise | 628 | 96.9% |
always-send crash classes (OOM, ANR, IllegalStateException) |
13 | 2.0% |
| deduped / rate-capped (the 401 storm) | 7 | 1.1% |
Upper bound of surviving volume: 2,750/month × 3.1% ≈ 87/month, ~17× under the 1,500/month target, and that ignores dedup and the hourly cap, which only push it lower. Crash classes still come through — the test fails if any of them stops being allowlisted, because hitting the number by silencing real crashes is a failure, not a win.
The one piece of evidence available on release day: check the artifact
Everything above is worthless if a build ships without the gate. That failure is
invisible in Sentry: while the org is over quota nothing is stored, so release tags stop
updating (opencode-mobile's stop at 0.4.12 / 2026-08-08 while clients keep submitting
~4.7/h) and a release:0.4.14 query returns an empty result that reads exactly like "no
errors from the new build".
The binary is checkable the same day. Hermes bytecode keeps string literals, so
scripts/verify-release-bundle.mjs greps base/assets/index.android.bundle inside the AAB
for the gate's own reason codes, the transport drop-list regex, and the
noise.dropped_since_last tag that only applyNoiseGate() writes — plus the baked-in DSN,
because a release built without EXPO_PUBLIC_SENTRY_DSN makes Sentry.init() a silent
no-op. It runs in publish-play-store.yml before the Play upload step, so a gateless
build cannot reach users.
It discriminates — this is not a self-confirming assertion:
| Artifact | Result |
|---|---|
v0.4.14 AAB, versionCode 151 (the build now on Play production, run 31807432647) |
passes, all six markers, DSN = project 4511436292292608 |
v0.4.13 AAB (run 31786473735, pre-gate) |
fails, all six markers absent |
Rejected: persisting gate state across launches
The rate caps (maxPerHour, maxNewPerHour) and the 6h per-fingerprint cooldown live in
process memory, so every cold start resets them. That looks like a hole worth plugging
with AsyncStorage; the session data says it is not.
Sentry session envelopes for opencode-mobile, 7d to 2026-08-14: 2,633 (267–541/day),
which is the install base's app-start rate. Persisting only pays off if a device launches
the app more often than the cooldown expires — i.e. if 376 launches/day ÷ devices > 4,
so only below ~94 active devices. One issue alone (connect timeout) has 104 distinct
users over 90d, so the install base is above that line and the two rates are within
rounding of each other. Persisting would add native storage I/O on the crash path to buy
nothing measurable. Revisit only if the app-start rate rises well above ~4/device/day.
(Those session envelopes are 100% client_discard, 96% network_error — sent at cold
start and at process teardown, when the transport often can't complete. Sessions are not
billed, so this costs no quota, but it does mean release health is not a usable signal for
this app either.)
Disclosure surfaces (must stay in sync)
| Surface | File |
|---|---|
| First-launch consent modal | src/components/TelemetryConsentModal.tsx |
| Settings toggle label/description | app/(tabs)/settings.tsx |
| Privacy policy (canonical md) | distribution/privacy-policy.md §3a, §3b, §4, §5 |
| Privacy policy (store/site html) | distribution/privacy-policy.html, docs/privacy/index.html (live gh-pages) |
| Play Data safety draft | distribution/play-listing.md |
| Play ops checklist | docs/playstore.md item 7 |
| Apple nutrition label | Apple addendum in distribution/privacy-policy.md (Usage Data → Product Interaction: Yes) |
Verification checklist — TODO
Not yet verified end-to-end. Each item needs a real device/emulator run with a network sniffer or PostHog live-events view:
- TODO: Fresh install → decline consent → exercise full app flow → confirm zero requests to
eu.i.posthog.comandsentry.io. - TODO: Fresh install → allow consent → confirm
app_openedarrives withis_first_open=true; second launch sendsis_first_open=false. - TODO: Onboarding quick-connect success path emits
connection_form_submitted(mode=quick)→connection_attempted(source=onboarding)→connection_succeeded. - TODO: Failure path emits
connection_failedwith a coarseerror_classand no raw error text/hostname in the payload. - TODO: Send message + receive response emits
message_sentandresponse_received; aborted run emits noresponse_received. - TODO: Revoke mid-session while offline (events buffered) → go online → confirm buffered events are dropped (no PostHog traffic after revoke).
- TODO: Revoke → re-grant in same session →
app_openednot double-counted. - TODO: Build without
EXPO_PUBLIC_POSTHOG_KEY→ analytics is a complete no-op (no init log, no network). - TODO: Inspect one real payload of every event type in PostHog and confirm property allowlist matches the schema table above.
- TODO: Play Console Data safety form re-submitted to match
distribution/play-listing.mddraft before next release.