d31afc0389
The gate ships inside the app binary, so it only runs on devices that took
v0.4.14. Grading it on a raw event count is a measurement error in both
directions: a still-high number at 30% uptake is the gate WORKING (~70% of
baseline is the model's own prediction), and a dip from a quiet weekend is not
efficacy. The scheduled re-reads on 08-17 / 08-21 / 09-05 would have hit the
first one first.
noise-gate-report.mjs folds Play's version share into the comparison
expected_post = baseline x (1 - gated_share x 0.969)
where 0.969 is measured, not guessed (90d replay in sentry-noise.test.ts), and
grades the measured rate against that instead of against the 100%-uptake
endpoint. It reports the endpoint separately, so 'is it on track today' and
'will it clear the 3,500/mo org gate' stop being the same question, and it
inverts the model to print the IMPLIED on-device efficacy so the constant is
checked rather than trusted.
It refuses to grade two windows that look like results but are not: no
client_discard/before_send (nothing ran the gate) and 0% Play share. Absence of
evidence gets its own verdict, UNGRADED.
Runs in CI because neither credential (Sentry org token, Play service account)
exists outside GitHub Secrets — an agent picking up the 08-21 read locally is
stuck otherwise. Weekly cron records the trend regardless.
Verified against the live org: 1.2h after the production rollout it reads
UNGRADED, before_send=0, 4.95/h vs a 4.71/h baseline — which is exactly right,
no device has the build yet.
Refs AGE-105
Co-authored-by: engineer <engineer@macbookpro.lan>
89 lines
3.4 KiB
YAML
89 lines
3.4 KiB
YAML
name: Sentry noise-gate report
|
|
|
|
# Grades the AGE-105 noise gate against the number it promised, with install-base
|
|
# uptake folded in — see scripts/noise-gate-report.mjs for why a raw event count
|
|
# cannot grade a gate that ships inside an app binary.
|
|
#
|
|
# It lives in CI rather than on a laptop for one blunt reason: neither credential
|
|
# it needs (Sentry org read token, Play service account) exists outside GitHub
|
|
# Secrets, so an agent picking up this measurement locally is stuck. Dispatch this
|
|
# instead and read the run summary:
|
|
#
|
|
# gh workflow run "Sentry noise-gate report" -f post=2026-08-21T00:00:00Z..now
|
|
# gh run watch <id> && gh run view <id> # table is in the job summary
|
|
#
|
|
# The scheduled Monday run exists so the trend is recorded even if nobody asks.
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
pre:
|
|
description: "Baseline window START..END (rate before the rollout reached devices)"
|
|
required: false
|
|
# Post-box-bot-fix, pre-mobile-rollout. The only clean baseline that
|
|
# measures mobile alone: AGE-55 silenced openclaw-box-bot at 06:19Z and
|
|
# v0.4.14 reached production at 14:22Z on the same day.
|
|
default: "2026-08-14T07:00:00Z..2026-08-14T14:00:00Z"
|
|
post:
|
|
description: "Measured window START..END (default: trailing 7d)"
|
|
required: false
|
|
default: ""
|
|
project:
|
|
description: "Sentry project slug"
|
|
required: false
|
|
default: "opencode-mobile"
|
|
schedule:
|
|
# Mondays 15:00 UTC. Weekly, not daily: a window under ~24h ranks sources but
|
|
# cannot certify a monthly rate, and Play vitals land with a multi-day lag.
|
|
- cron: "0 15 * * 1"
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: sentry-noise-gate-report-${{ github.ref }}
|
|
cancel-in-progress: false
|
|
|
|
jobs:
|
|
report:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
- uses: actions/setup-node@v6
|
|
with:
|
|
node-version: 24
|
|
|
|
# The grading model is the part that can be wrong silently, so it is tested
|
|
# before it is trusted — in the same job that publishes the number.
|
|
- name: Test the grading model
|
|
run: node --test scripts/noise-gate-report.test.mjs
|
|
|
|
- name: Report
|
|
env:
|
|
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_PRODUCT_INTELLIGENCE_TOKEN || secrets.SENTRY_AUTH_TOKEN }}
|
|
SENTRY_ORG: ${{ secrets.SENTRY_ORG || 'vibetechnologies' }}
|
|
GOOGLE_PLAY_SERVICE_ACCOUNT_JSON: ${{ secrets.PLAY_STORE_SERVICE_ACCOUNT_JSON }}
|
|
run: |
|
|
set -euo pipefail
|
|
args=(--pre "${{ inputs.pre || '2026-08-14T07:00:00Z..2026-08-14T14:00:00Z' }}")
|
|
args+=(--project "${{ inputs.project || 'opencode-mobile' }}")
|
|
if [ -n "${{ inputs.post }}" ]; then args+=(--post "${{ inputs.post }}"); fi
|
|
node scripts/noise-gate-report.mjs "${args[@]}"
|
|
|
|
- name: Raw org volume (per project, per outcome, with client_discard reasons)
|
|
if: always()
|
|
env:
|
|
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_PRODUCT_INTELLIGENCE_TOKEN || secrets.SENTRY_AUTH_TOKEN }}
|
|
SENTRY_ORG: ${{ secrets.SENTRY_ORG || 'vibetechnologies' }}
|
|
run: |
|
|
set -euo pipefail
|
|
{
|
|
echo ''
|
|
echo '### Raw org volume'
|
|
echo '```'
|
|
node scripts/sentry-volume-report.mjs --by-reason || true
|
|
echo '```'
|
|
} >> "$GITHUB_STEP_SUMMARY"
|