test(ci): widen cua-smoke gate to real core journey (connect→send→reply→list)
The CI opencode server had NO LLM provider configured — the server log only showed "listening", never a model. opencode-ai (released npm pkg) does not read AZURE_OPENAI_* for its own LLM; it needs an explicit provider in opencode.json + a default `model`. So send_message/multi_turn could never pass and the gate was stuck on --only-connect-scenario (UI journey minus the model reply). - Wire opencode to the same Azure resource the CUA driver uses via a generated ~/.config/opencode/opencode.json (@ai-sdk/azure provider, resourceName derived from the endpoint secret at runtime, apiKey from env, default model azure/gpt-5.4). - Add a deterministic REST probe step: create a session + send a prompt and check for an assistant reply BEFORE the ~30min emulator run, exporting MODEL_CAPABLE. - Add --scenarios to android-cua-smoke.py to run an explicit named set. - Emulator step now runs connect_and_verify_sessions + send_message + verify_session_list when MODEL_CAPABLE=true; falls back to the UI-only journey (connect + verify_session_list) otherwise, logging the environmental reason. - Raise --max-steps to 40 so multiple scenarios fit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -634,6 +634,13 @@ def main():
|
||||
help="Run ONLY the connect-and-verify-sessions scenario. Use in CI with a "
|
||||
"local opencode server for a deterministic true-E2E (no model backend needed).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--scenarios",
|
||||
help="Comma-separated explicit scenario set to run, e.g. "
|
||||
"'connect_and_verify_sessions,send_message,verify_session_list'. "
|
||||
"Valid names: connect_and_verify_sessions, send_message, multi_turn, "
|
||||
"verify_session_list. Overrides --only-connect-scenario and the default set.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Verify ADB
|
||||
@@ -650,7 +657,18 @@ def main():
|
||||
"goal": _connect_and_verify_sessions_goal(connect_url),
|
||||
}
|
||||
|
||||
if args.only_connect_scenario:
|
||||
if args.scenarios:
|
||||
# Explicit named set (CI widened gate). Look up by name across the full catalog.
|
||||
catalog = {connect_scenario["name"]: connect_scenario}
|
||||
for s in SMOKE_SCENARIOS:
|
||||
catalog[s["name"]] = s
|
||||
requested = [n.strip() for n in args.scenarios.split(",") if n.strip()]
|
||||
unknown = [n for n in requested if n not in catalog]
|
||||
if unknown:
|
||||
sys.exit(f"Unknown scenario(s): {', '.join(unknown)}. "
|
||||
f"Valid: {', '.join(catalog.keys())}")
|
||||
scenarios = [catalog[n] for n in requested]
|
||||
elif args.only_connect_scenario:
|
||||
# CI true-E2E: just connect to the local opencode server and verify the list.
|
||||
scenarios = [connect_scenario]
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user