Files
opencode-mobile/distribution/waitlist-signup-path-coverage.md
Den 4919164bca docs(waitlist): make the AGE-100 after-number a grep, not an inbox crawl (#168)
* docs(waitlist): make the AGE-100 after-number a grep, not an inbox crawl

The doc told a future reader to split recovered mailto signups "by the App:
line", which in practice meant opening Chatwoot conversations one by one and
eyeballing bodies — slow, and easy to get wrong in the direction that matters
(missing a stamped mail reads as "clean").

The reconciler now does the split itself
(VibeBrowserProductPage#237) and prints it:

  scanned=N synced=N skipped=N failed=N unstamped=N stamped=N builds=vX:N

Document that line as the read, with the exact gh commands, and state the pass
condition explicitly: stamped must be 0, because stamped>0 means a build that
carries the AGE-87 retry queue still fell through to mailto.

Refs AGE-100.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

* docs(waitlist): record the first post-release reading of the build split

Proof the measurement pipeline works against the real inbox, not just a stub:
the first reconciler run carrying the split (2026-08-14 09:57 UTC) printed

  scanned=34 synced=0 skipped=34 failed=0 unstamped=0 stamped=0

— the same 34 known conversations as the pre-release baseline. Labelled as one
hour of exposure rather than a verdict, so nobody mistakes it for the week-out
number that actually closes AGE-100.

Refs AGE-100.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

---------

Co-authored-by: engineer <engineer@macbookpro.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-14 03:21:57 -07:00

6.6 KiB
Raw Blame History

Waitlist signup-path coverage — how much of the install base can't sign up in-app

Measured 2026-08-14 for AGE-61. Re-run with node scripts/play-version-share.mjs --github.

The question

The in-app OpenCode Connect waitlist form posts to POST /api/beta-signup (Brevo list 4). That code path first shipped in v0.4.8 (commit 0fdfb54, 2026-07-18). Every older build has exactly one path: open a mailto: to support@agentlabs.cc, which lands in a human inbox and nowhere near the waitlist store. 20 of 21 signups between 2026-08-03 and 2026-08-13 were lost that way. So: how many people are still on a build with no working signup path?

Answer, by channel

Channel Population measured Pre-v0.4.8 share
Google Play (auto-updates) ~90–100 daily distinct users, 2026-07-31 → 2026-08-13 0% — Play reports a single versionCode, 142 (v0.4.10). Any older build is below Play's reporting floor (<10 users/day).
Sideload — GitHub release APKs (never auto-update) 1,682 lifetime APK downloads across all releases 25.9% (436 downloads) are pre-v0.4.8 builds.

There is no iOS App Store listing (itunes.apple.com/lookup?bundleId=cc.agentlabs.opencode returns 0 results) and the app is not on IzzyOnDroid, so those channels contribute nothing.

What that means

  1. Play is not the problem. Auto-update did its job: the measurable Android active base is effectively 100% on a build that can sign up through the API.
  2. The stale cohort is sideload-only and permanent. ~436 devices pulled an APK that predates the signup API. Nothing will ever update them; they will keep emitting mailto signups until the owner manually re-downloads. The hourly reconciler (VibeBrowserProductPage/.github/workflows/waitlist-mailto-reconcile.yml) is what keeps those signups from being lost, and it is not a temporary measure.
  3. Stale builds are not the only source of mailto signups — they are, as of AGE-87, the only remaining one. Until v0.4.12 the fallback also fired on network error, timeout (8s) and 5xx, so a user on a current build with flaky mobile data took the same lossy path. That path is gone: a failed signup is now persisted on-device (opencode.waitlist.pending.v1, AsyncStorage) and retried on every app foreground (src/lib/waitlist.ts queue section, flushed from app/_layout.tsx). mailto: is only ever opened by an explicit user tap after repeated retry failures. Expect the reconciler's synced_count to trend toward the sideload cohort only. Any plan that assumes "ship an update and the leak closes" is still wrong for those ~436 devices.

After-number: how it gets attributed (AGE-100)

Pending — v0.4.13 (the first build users can actually run the retry queue on) went to the Play production track on 2026-08-14 as versionCode 149 (run 31786473735). Re-read one week out.

Baseline at release time, from the hourly reconciler's last run before the cut (Waitlist Mailto Reconcile, 2026-08-14 07:59 UTC): scanned=34 synced=0 skipped=34 failed=0 — i.e. 34 known mailto conversations, all already healed into the store, nothing new that hour.

The reconciler reads a Chatwoot mail body, so "split by app version" only works if the body carries one. It did not. As of v0.4.13 the escape hatch stamps App: OpenCode Mobile v<version> into the mail (buildWaitlistMailtoUrl, src/lib/waitlist.ts). That gives a clean read a week out:

Mail body Means
no App: line a build older than v0.4.13 — expected, this is the ~436-device sideload cohort no release can reach
App: OpenCode Mobile v0.4.13 or newer a current build still reached mailto — the retry queue leaked, file it as a new defect with the mail as evidence

Reading the after-number

Do not open Chatwoot conversations by hand — the reconciler does the split itself as of VibeBrowserProductPage#237. Every hourly run now prints one grep-able line:

scanned=N synced=N skipped=N failed=N unstamped=N stamped=N builds=v0.4.13:N
  • unstamped — recovered signups from builds older than v0.4.13. Expected; this is the cohort no shipped code can reach, and it is the number that should account for essentially all of synced.
  • stamped — recovered signups from builds that carry the AGE-87 retry queue and fell back anyway. This must be 0. Anything above 0 is a live defect, and the reconciler already says so in its alert issue and in the Chatwoot internal note, naming the exact conversations.

So the week-out read is:

gh run list --repo dzianisv/VibeBrowserProductPage \
  --workflow "Waitlist Mailto Reconcile" --limit 200 \
  --json databaseId,createdAt,conclusion
# then, per run id:
gh run view --repo dzianisv/VibeBrowserProductPage <id> --log | grep -E "^scanned="

Sum stamped across the week. stamped=0 closes AGE-100 with the number; stamped>0 opens a defect with those conversations as evidence.

First reading after the release, from the run that first carried the split (run 31790218194, 2026-08-14 09:57 UTC, against the real inbox):

scanned=34 synced=0 skipped=34 failed=0 unstamped=0 stamped=0

Same 34 known conversations as the pre-release baseline, nothing new. That is one hour of exposure, not a verdict — it only proves the measurement pipeline reports the split end to end. The week-out sum is what closes AGE-100.

Method / reproducing

GOOGLE_PLAY_SERVICE_ACCOUNT_JSON="$(cat play-store-key.json)" \
  node scripts/play-version-share.mjs --days 14 --github
  • Play numbers come from the Play Developer Reporting API, crashRateMetricSet → distinctUsers grouped by versionCode. That is Play's own vitals denominator: users who actually opened the app. It only covers devices with usage-and-diagnostics sharing on, and Play rounds it — which is why we quote a share, never an absolute install count.
  • Play versionCodes are not the ones in android/app/build.gradle: the publish workflow overwrites them with github.run_number + 100. The mapping in the script was derived from the successful runs of publish-play-store.yml (versionCode 139 = v0.4.8 = first build with the API path).
  • The service account is the same PLAY_STORE_SERVICE_ACCOUNT_JSON already used to publish; the script needs only the playdeveloperreporting scope and mints its own token, no extra deps.