test(ci): widen cua-smoke gate to real core journey (connect→send→reply→list)

The CI opencode server had NO LLM provider configured — the server log only
showed "listening", never a model. opencode-ai (released npm pkg) does not read
AZURE_OPENAI_* for its own LLM; it needs an explicit provider in opencode.json +
a default `model`. So send_message/multi_turn could never pass and the gate was
stuck on --only-connect-scenario (UI journey minus the model reply).

- Wire opencode to the same Azure resource the CUA driver uses via a generated
  ~/.config/opencode/opencode.json (@ai-sdk/azure provider, resourceName derived
  from the endpoint secret at runtime, apiKey from env, default model azure/gpt-5.4).
- Add a deterministic REST probe step: create a session + send a prompt and check
  for an assistant reply BEFORE the ~30min emulator run, exporting MODEL_CAPABLE.
- Add --scenarios to android-cua-smoke.py to run an explicit named set.
- Emulator step now runs connect_and_verify_sessions + send_message +
  verify_session_list when MODEL_CAPABLE=true; falls back to the UI-only journey
  (connect + verify_session_list) otherwise, logging the environmental reason.
- Raise --max-steps to 40 so multiple scenarios fit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
engineer
2026-06-08 05:58:19 -07:00
parent 42fa36f3cd
commit cfb0d9fe32
2 changed files with 95 additions and 4 deletions

View File

@@ -634,6 +634,13 @@ def main():
help="Run ONLY the connect-and-verify-sessions scenario. Use in CI with a "
"local opencode server for a deterministic true-E2E (no model backend needed).",
)
parser.add_argument(
"--scenarios",
help="Comma-separated explicit scenario set to run, e.g. "
"'connect_and_verify_sessions,send_message,verify_session_list'. "
"Valid names: connect_and_verify_sessions, send_message, multi_turn, "
"verify_session_list. Overrides --only-connect-scenario and the default set.",
)
args = parser.parse_args()
# Verify ADB
@@ -650,7 +657,18 @@ def main():
"goal": _connect_and_verify_sessions_goal(connect_url),
}
if args.only_connect_scenario:
if args.scenarios:
# Explicit named set (CI widened gate). Look up by name across the full catalog.
catalog = {connect_scenario["name"]: connect_scenario}
for s in SMOKE_SCENARIOS:
catalog[s["name"]] = s
requested = [n.strip() for n in args.scenarios.split(",") if n.strip()]
unknown = [n for n in requested if n not in catalog]
if unknown:
sys.exit(f"Unknown scenario(s): {', '.join(unknown)}. "
f"Valid: {', '.join(catalog.keys())}")
scenarios = [catalog[n] for n in requested]
elif args.only_connect_scenario:
# CI true-E2E: just connect to the local opencode server and verify the list.
scenarios = [connect_scenario]
else: