measure: how much of the install base has no in-app waitlist signup path (#164)

AGE-61 asked for a number: what share of installs is still on a build older
than v0.4.8, where the ONLY waitlist path is a mailto: to a human inbox that
nothing reconciles back into the store (20 of 21 signups lost, 2026-08-03 →
2026-08-13). Answering it by hand is how it stays unanswered next quarter, so
this is a script, not a screenshot.

- scripts/play-version-share.mjs: Play Developer Reporting API
  (crashRateMetricSet -> distinctUsers by versionCode) for the auto-updating
  channel, plus --github for lifetime release-APK downloads per tag, which is
  the only per-version signal the sideload channel emits. Mints its own token
  from the service account we already ship to CI; no new deps, no new secret.
  Play versionCodes are run_number+100, NOT the gradle ones — the mapping is
  derived from the publish runs and documented inline (139 = v0.4.8).
- distribution/waitlist-signup-path-coverage.md: the measured answer.

The answer, 2026-08-14: Play is 0% stale (single reported versionCode 142 =
v0.4.10, ~90-100 daily users); the sideload channel is 25.9% stale (436 of
1682 lifetime APK downloads predate v0.4.8) and can never auto-update. There
is no iOS listing and no IzzyOnDroid presence, so nothing else contributes.

Two consequences worth stating plainly: the hourly mailto reconciler is
permanent infrastructure, not a stopgap; and shipping updates does NOT close
the leak, because on current builds the fallback still fires on timeout/5xx/
offline (src/lib/waitlist.ts:shouldFallbackToMailto).

Co-authored-by: engineer <engineer@macbookpro.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Den
2026-08-14 00:42:42 -07:00
committed by GitHub
parent c92327570b
commit 98233d351f
2 changed files with 363 additions and 0 deletions

View File

@@ -0,0 +1,55 @@
# Waitlist signup-path coverage — how much of the install base can't sign up in-app
_Measured 2026-08-14 for [AGE-61]. Re-run with `node scripts/play-version-share.mjs --github`._
## The question
The in-app OpenCode Connect waitlist form posts to `POST /api/beta-signup` (Brevo list 4).
That code path first shipped in **v0.4.8** (commit `0fdfb54`, 2026-07-18). Every older build
has exactly one path: open a `mailto:` to support@agentlabs.cc, which lands in a human inbox
and nowhere near the waitlist store. 20 of 21 signups between 2026-08-03 and 2026-08-13 were
lost that way. So: **how many people are still on a build with no working signup path?**
## Answer, by channel
| Channel | Population measured | Pre-v0.4.8 share |
| --- | --- | --- |
| Google Play (auto-updates) | ~90–100 daily distinct users, 2026-07-31 → 2026-08-13 | **0%** — Play reports a single versionCode, `142` (v0.4.10). Any older build is below Play's reporting floor (<10 users/day). |
| Sideload — GitHub release APKs (never auto-update) | 1,682 lifetime APK downloads across all releases | **25.9%** (436 downloads) are pre-v0.4.8 builds. |
There is no iOS App Store listing (`itunes.apple.com/lookup?bundleId=cc.agentlabs.opencode`
returns 0 results) and the app is not on IzzyOnDroid, so those channels contribute nothing.
## What that means
1. **Play is not the problem.** Auto-update did its job: the measurable Android active base is
effectively 100% on a build that can sign up through the API.
2. **The stale cohort is sideload-only and permanent.** ~436 devices pulled an APK that predates
the signup API. Nothing will ever update them; they will keep emitting mailto signups until
the owner manually re-downloads. The hourly reconciler
(`VibeBrowserProductPage/.github/workflows/waitlist-mailto-reconcile.yml`) is what keeps those
signups from being lost, and it is not a temporary measure.
3. **Stale builds are not the only source of mailto signups.** In current builds the fallback also
fires on network error, timeout (8s) and 5xx (`src/lib/waitlist.ts:shouldFallbackToMailto`).
A user on v0.4.12 with flaky mobile data takes the same lossy path. Any plan that assumes
"ship an update and the leak closes" is wrong.
## Method / reproducing
```bash
GOOGLE_PLAY_SERVICE_ACCOUNT_JSON="$(cat play-store-key.json)" \
node scripts/play-version-share.mjs --days 14 --github
```
- Play numbers come from the Play Developer Reporting API, `crashRateMetricSet` →
`distinctUsers` grouped by `versionCode`. That is Play's own vitals denominator: users who
actually opened the app. It only covers devices with usage-and-diagnostics sharing on, and Play
rounds it — which is why we quote a share, never an absolute install count.
- Play versionCodes are **not** the ones in `android/app/build.gradle`: the publish workflow
overwrites them with `github.run_number + 100`. The mapping in the script was derived from the
successful runs of `publish-play-store.yml` (`versionCode 139` = v0.4.8 = first build with the
API path).
- The service account is the same `PLAY_STORE_SERVICE_ACCOUNT_JSON` already used to publish; the
script needs only the `playdeveloperreporting` scope and mints its own token, no extra deps.
[AGE-61]: https://github.com/dzianisv/VibeBrowserProductPage/pull/234