fix(e2e): stop asserting on SSE reply in activation-positive — closes #90 (#102)

* fix(e2e): stop asserting on SSE reply in activation-positive — CI-harness limitation, not a product bug (closes #90)

Extensive investigation (see PR #102 for the full trail) into "the
positive flow's assistant reply never renders" tried four independent
SSE client transports in src/lib/sdk.ts global.events(): the
already-shipped expo/fetch ReadableStream reader, a hand-rolled
XMLHttpRequest reader, react-native-sse, and react-native-fetch-api's
`reactNative: { textStreaming: true }`. Every one delivers exactly one
chunk right after connecting to the mock server and then nothing until
the connection closes, regardless of API choice or frame size (a ~4KB
padding experiment ruled out a buffer-size threshold).

A raw-socket probe (a plain BSD-sockets client with zero React Native
involvement, run via `adb shell` through the identical adb-reverse
tunnel the app uses) streamed every heartbeat from the mock server
incrementally in real time over the same connection. That rules out
adb-reverse and the mock's flush behavior and isolates the stall to
React Native Android's OkHttp-backed networking layer buffering a
long-lived streaming HTTP response in this specific Android-emulator +
Node-mock + adb-reverse combination — not a defect in any particular
client library.

There's no evidence this reproduces against a real opencode server on
a real device/network: issue #76's 65 affected users prove real SSE
connections stream live agent output in production (the bug they hit
was the 401-retry storm, not a missing reply). Since expo/fetch is the
already-shipped, production-proven transport and none of the
alternatives showed any advantage in this harness, the transport stays
unchanged.

What changes instead: .maestro/flows/activation-positive.yaml no
longer waits on the SSE-streamed reply, since asserting on it here
would assert on a CI-harness limitation, not real app behavior. It now
verifies everything reliably observable — consent, connect, session
creation, and the optimistic local echo of the sent message — and
activation-e2e.yml's `continue-on-error: true` (added because this
suite had never passed) comes off, so it blocks PRs on regressions in
what it does cover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(e2e): repair stale "401" assertion in activation-negative-401 (refs #90)

Removing activation-e2e.yml's continue-on-error surfaced a second, unrelated
stale assertion once the suite was actually enforcing again: the negative
flow's connect-time-401 case asserts a literal "401" that PR #79 (401/403
auth-stop handling) and #103 (i18n) apparently moved out of what's
rendered — "Connection Failed" still passes, "401" now fails.

The alert body interpolates two pieces: probeConnection()'s summary (which
turns out to be misclassified as "connection actually works now" for this
case — diagnostics-classify.ts's `health.ok` only reflects "fetch() didn't
throw", not HTTP status, a separate real bug, out of scope for this PR) and
testConnection()'s caught error message, which is sdk.ts's
apiErrorFor(401, ...) text and always contains the mock's
`{"error":"Unauthorized",...}` body per src/lib/api-error.test.ts. Swapped
the assertion to "Unauthorized" and added a temporary console.log of both
pieces in app/connection/add.tsx to confirm exactly what renders from CI
logcat (removed once confirmed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(e2e): assert alert action buttons, not body text — native AlertDialog body isn't in Maestro's a11y tree (refs #90)

The diagnostic added last commit confirmed the Connection Failed alert's
body DOES contain the real error ("API Error: 401 -
{\"error\":\"Unauthorized\",...}", via logcat: '[connect] failure alert
content' logged both the (separately buggy, out-of-scope)
probeConnection summary and the correct testConnection error text). Yet
both "401" and "Unauthorized" assertions still failed against the same
on-screen alert. That means Maestro's accessibility-tree text matching
on this Android AlertDialog only sees the title, not the message body —
so no substring of the body was ever going to match.

Switched to asserting what's actually reachable: the title "Connection
Failed" (unchanged, already passing) plus both action button labels,
"OK" and "Share report" (src/lib/i18n/en.json common.ok /
common.shareReport). That still proves the test's real intent — a
visible, actionable error with a dismiss and a share-report path, never
a silent failure (issue #76) — using strings actually present in the
accessibility tree instead of guessing at unreachable body text.

Removes the temporary console.log diagnostic from
app/connection/add.tsx now that its purpose (confirming exactly what
renders) is done.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(e2e): split activation-e2e into blocking core + non-blocking newer flows (refs #90, refs #104)

With this PR's fixes, activation-positive and activation-negative-401
(the coverage issue #90 actually scoped) now run green — but removing
activation-e2e.yml's continue-on-error surfaced that four flows added
after the initial suite (#82's directory-picker/all-sessions/
variant-picker, #101's diff-scroll) have never once run to completion
in CI: they always sat behind whichever activation flow failed first,
so they were merged and have run unverified against the current
UI/mock this whole time. directory-picker fails immediately at
`id: directory-row-frontend`; the other three are untriaged.

Fixing four separate, previously-never-green UI surfaces is out of
#90's scope and unbounded in this PR. scripts/run-e2e-flows.sh now
splits the flow list into CORE_FLOWS (the two #90 covers — blocking,
fails the job on a regression) and NEWER_FLOWS (the four newer ones —
always run, each one's pass/fail reported via echo/::warning::, but
never fails the job). This lets activation-e2e.yml enforce the
activation coverage that's now verified, without either leaving it red
forever or spending unbounded time inside this PR chasing four
unrelated UI surfaces.

Filed #104 to track hardening each NEWER flow and moving it back into
CORE_FLOWS once confirmed green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Den
2026-07-17 14:52:37 -07:00
committed by GitHub
parent f945cab5c6
commit 5822e471c6
4 changed files with 109 additions and 29 deletions

View File

@@ -11,7 +11,23 @@ set -uo pipefail
ROOT="$(pwd)" # capture BEFORE any cd, so diag paths are absolute
APK="android/app/build/outputs/apk/release/app-release.apk"
FLOWS=(activation-positive activation-negative-401 directory-picker all-sessions variant-picker diff-scroll)
# CORE = the activation coverage issue #90 verified green (consent -> connect
# -> send, and the connect-time-401 visible-error case). A failure here fails
# the job — this is the suite's actual regression gate.
#
# NEWER = flows added after the initial activation suite (#82's
# directory-picker/all-sessions/variant-picker, #101's diff-scroll) that have
# never once run to completion in CI: they always sat behind the positive
# flow's stale SSE-reply assertion (issue #90 mode B, now fixed) or the
# negative-401 flow's stale "401" assertion (also now fixed), both of which
# made the job fail before reaching them — so they were merged and have been
# running unverified against the current UI/mock ever since. Run them for
# visibility (each flow's pass/fail is reported below) but don't block on
# them yet — see issue #104 for hardening them (fixing whatever stale
# selectors/seeding surface) and moving each back into CORE once confirmed
# green.
CORE_FLOWS=(activation-positive activation-negative-401)
NEWER_FLOWS=(directory-picker all-sessions variant-picker diff-scroll)
mkdir -p "$ROOT/artifacts/screenshots" "$ROOT/artifacts/diag"
echo "== installing APK =="
@@ -85,12 +101,26 @@ trap dump_diag EXIT
cd "$ROOT/artifacts/screenshots"
rc=0
for f in "${FLOWS[@]}"; do
echo "--- flow: $f ---"
for f in "${CORE_FLOWS[@]}"; do
echo "--- flow (core, blocking): $f ---"
if ! maestro test --debug-output "$ROOT/artifacts/diag/maestro-$f" "$ROOT/.maestro/flows/$f.yaml"; then
echo "::error::Maestro flow failed: $f"
rc=1
break
fi
done
# Newer flows: always run all of them (no `break` on failure) and never
# affect $rc — see the CORE_FLOWS/NEWER_FLOWS comment above for why. Each
# flow's own pass/fail is still clearly reported, just non-blocking.
echo "== newer flows (non-blocking, tracked for hardening) =="
for f in "${NEWER_FLOWS[@]}"; do
echo "--- flow (newer, non-blocking): $f ---"
if maestro test --debug-output "$ROOT/artifacts/diag/maestro-$f" "$ROOT/.maestro/flows/$f.yaml"; then
echo "== newer flow PASSED: $f =="
else
echo "::warning::newer flow FAILED (non-blocking): $f"
fi
done
exit $rc