fix(metrics): repair review triage — correct secret wiring, privacy-safe aggregated issues. Closes #61. Refs #60. (#78)
* fix(metrics): repair review triage — correct secret wiring, privacy-safe aggregated issues - triage-reviews.yml read secrets.GOOGLE_SERVICE_ACCOUNT_JSON, which doesn't exist; map the real PLAY_STORE_SERVICE_ACCOUNT_JSON secret onto the env var the script expects. - triage-reviews.py rewritten to maintain a single sanitized, deduped "Play Store Review Triage" issue instead of one public issue per review. The old version leaked reviewer full names and verbatim review text into public GitHub issues and spammed the tracker. The new version aggregates actionable (<=3 star) reviews into one issue with rating counts, a word-frequency theme summary (no quoted sentences), and opaque review_id references for Play Console lookup. An embedded HTML comment marker (matching the product-intelligence.mjs pattern) holds the current actionable review_id set so runs update in place and skip entirely when nothing changed. - product-intelligence.yml referenced the nonexistent SENTRY_PRODUCT_INTELLIGENCE_TOKEN secret, causing the daily cron to fail silently (#60). Fall back to SENTRY_AUTH_TOKEN when the dedicated read-only token isn't configured. - docs/playstore.md: document that Play Console is still the only trusted source for acquisition/uninstall metrics (product-intelligence.mjs defers this), and that review-based signals are sourced via the Android Publisher API through PLAY_STORE_SERVICE_ACCOUNT_JSON. Closes #61. Refs #60. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJKAQ6HAikWGQK7PGZ5Y4E * fix(triage): fail visibly when GOOGLE_SERVICE_ACCOUNT_JSON is missing Review finding on PR #78: env_client() exited 0 on missing credentials, so the scheduled workflow would report success while silently doing nothing — contradicting issue #61's 'missing credentials fail visibly' done-criteria. --------- Co-authored-by: engineer <engineer@gray-knight-m1.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Den
5
.github/workflows/product-intelligence.yml
vendored
5
.github/workflows/product-intelligence.yml
vendored
@@ -31,7 +31,10 @@ jobs:
|
||||
continue-on-error: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_PRODUCT_INTELLIGENCE_TOKEN }}
|
||||
# SENTRY_PRODUCT_INTELLIGENCE_TOKEN is a dedicated read-only token and is
|
||||
# preferred if/when it exists; falls back to the general-purpose
|
||||
# SENTRY_AUTH_TOKEN so the daily cron doesn't silently no-op (#60).
|
||||
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_PRODUCT_INTELLIGENCE_TOKEN || secrets.SENTRY_AUTH_TOKEN }}
|
||||
SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
|
||||
SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
|
||||
run: |
|
||||
|
||||
4
.github/workflows/triage-reviews.yml
vendored
4
.github/workflows/triage-reviews.yml
vendored
@@ -30,6 +30,8 @@ jobs:
|
||||
|
||||
- name: Triage Play Store reviews
|
||||
env:
|
||||
GOOGLE_SERVICE_ACCOUNT_JSON: ${{ secrets.GOOGLE_SERVICE_ACCOUNT_JSON }}
|
||||
# Repo secret is named PLAY_STORE_SERVICE_ACCOUNT_JSON (see docs/playstore.md);
|
||||
# mapped here to the env var name the script expects.
|
||||
GOOGLE_SERVICE_ACCOUNT_JSON: ${{ secrets.PLAY_STORE_SERVICE_ACCOUNT_JSON }}
|
||||
GH_TOKEN: ${{ secrets.GH_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
run: python scripts/triage-reviews.py
|
||||
|
||||
@@ -97,6 +97,27 @@ After 14 days on Closed testing with 12+ active testers → promote to Productio
|
||||
|
||||
---
|
||||
|
||||
## Acquisition metrics — trusted source
|
||||
|
||||
There is currently no verified, least-privilege source wired up for Play
|
||||
Store acquisition/uninstall metrics (installs, uninstalls, store listing
|
||||
conversion). `scripts/product-intelligence.mjs` explicitly lists this as a
|
||||
**deferred** metric until a proper reporting contract exists — do not treat
|
||||
ad-hoc Play Console screenshots or manual exports as a trusted feed for
|
||||
automated reporting.
|
||||
|
||||
Until that's implemented, the Play Console UI
|
||||
(https://play.google.com/console/u/2/developers/8842655543970815326) is the
|
||||
only source of truth for acquisition numbers, checked manually. Review
|
||||
volume/rating triage (a related but separate signal) is automated via
|
||||
`.github/workflows/triage-reviews.yml` + `scripts/triage-reviews.py`, which
|
||||
reads reviews through the Android Publisher API using the
|
||||
`PLAY_STORE_SERVICE_ACCOUNT_JSON` GitHub secret (see "Linked GCP resources"
|
||||
above) — this is the trusted source for review-based signals, not any
|
||||
scraped or manually copied review text.
|
||||
|
||||
---
|
||||
|
||||
## Sibling channels: F-Droid + IzzyOnDroid
|
||||
|
||||
OpenCode Mobile is also distributed via F-Droid (mainline) and IzzyOnDroid —
|
||||
|
||||
@@ -2,9 +2,21 @@
|
||||
"""
|
||||
Play Store review triage script.
|
||||
|
||||
Fetches recent Play Store reviews via the Android Publisher API and creates
|
||||
GitHub issues for low-rated or bug-report reviews so they enter the normal
|
||||
fix cycle without manual monitoring.
|
||||
Fetches recent Play Store reviews via the Android Publisher API and maintains
|
||||
a single, sanitized, aggregated GitHub issue summarizing low-rated reviews so
|
||||
they enter the normal fix cycle without manual monitoring.
|
||||
|
||||
Privacy: the created/updated issue is public, so it never contains a
|
||||
reviewer's name or verbatim review text. It only contains star-rating
|
||||
counts, a paraphrased common-terms summary derived from word frequency
|
||||
(not quoted sentences), and opaque review_id references an operator can
|
||||
look up in the Play Console.
|
||||
|
||||
Dedup: an HTML comment marker embeds the sorted set of review_ids
|
||||
represented in the issue (same pattern as scripts/product-intelligence.mjs).
|
||||
If the current run's actionable review_id set is unchanged from the
|
||||
marker, the run skips without touching the issue. Otherwise it
|
||||
creates the issue (first run) or updates it in place.
|
||||
|
||||
Required env vars:
|
||||
GOOGLE_SERVICE_ACCOUNT_JSON — full JSON key for the service account
|
||||
@@ -15,12 +27,13 @@ Required env vars:
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import tempfile
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
import google.auth
|
||||
from google.oauth2 import service_account
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
@@ -29,14 +42,30 @@ PACKAGE_NAME = os.environ.get("PACKAGE_NAME", "cc.agentlabs.opencode")
|
||||
DAYS_BACK = int(os.environ.get("DAYS_BACK", "7"))
|
||||
GH_TOKEN = os.environ.get("GH_TOKEN", "")
|
||||
|
||||
ISSUE_TITLE = "Play Store Review Triage"
|
||||
MARKER_RE = re.compile(r"<!-- review-triage:([^>]*?) -->")
|
||||
|
||||
SCOPES = ["https://www.googleapis.com/auth/androidpublisher"]
|
||||
|
||||
STOPWORDS = {
|
||||
"the", "and", "for", "that", "this", "with", "have", "has", "not",
|
||||
"you", "your", "but", "app", "when", "just", "very", "also", "from",
|
||||
"are", "was", "were", "its", "it's", "can't", "cant", "dont", "don't",
|
||||
"wont", "won't", "them", "they", "their", "there", "here", "what",
|
||||
"would", "could", "should", "been", "being", "than", "then", "into",
|
||||
"about", "even", "still", "again", "after", "before", "which", "some",
|
||||
"only", "more", "most", "much", "many", "will", "does", "did", "how",
|
||||
"why", "get", "got", "use", "used", "using", "please", "make", "made",
|
||||
}
|
||||
|
||||
def get_service():
|
||||
|
||||
def env_client():
|
||||
raw = os.environ.get("GOOGLE_SERVICE_ACCOUNT_JSON", "")
|
||||
if not raw:
|
||||
print("GOOGLE_SERVICE_ACCOUNT_JSON not set — skipping.")
|
||||
sys.exit(0)
|
||||
# Fail visibly (issue #61 done-criteria): a missing credential must
|
||||
# turn the workflow run red, not report success while doing nothing.
|
||||
print("ERROR: GOOGLE_SERVICE_ACCOUNT_JSON not set — cannot fetch reviews.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
info = json.loads(raw)
|
||||
creds = service_account.Credentials.from_service_account_info(info, scopes=SCOPES)
|
||||
return build("androidpublisher", "v3", credentials=creds, cache_discovery=False)
|
||||
@@ -62,8 +91,9 @@ def fetch_reviews(service):
|
||||
continue
|
||||
results.append({
|
||||
"review_id": review.get("reviewId", ""),
|
||||
"author": review.get("authorName", "anonymous"),
|
||||
"rating": comment.get("starRating", 0),
|
||||
# author name is intentionally not carried past this point —
|
||||
# it never enters the aggregated public issue.
|
||||
"rating": int(comment.get("starRating", 0) or 0),
|
||||
"text": comment.get("text", ""),
|
||||
"date": dt.strftime("%Y-%m-%d"),
|
||||
"lang": comment.get("reviewerLanguage", "en"),
|
||||
@@ -74,81 +104,158 @@ def fetch_reviews(service):
|
||||
return results
|
||||
|
||||
|
||||
def issue_exists(title_prefix: str) -> bool:
|
||||
"""Check if a GitHub issue with this title prefix already exists."""
|
||||
def common_terms(reviews, top_n=8):
|
||||
"""Word-frequency summary across review text. Deliberately NOT a
|
||||
verbatim excerpt — single lowercased tokens only, no sentence
|
||||
structure, no attribution to a specific review or author."""
|
||||
counts = Counter()
|
||||
for review in reviews:
|
||||
for word in re.findall(r"[a-zA-Z']{4,}", review["text"].lower()):
|
||||
if word in STOPWORDS:
|
||||
continue
|
||||
counts[word] += 1
|
||||
return [word for word, _ in counts.most_common(top_n)]
|
||||
|
||||
|
||||
def find_existing_issue():
|
||||
result = subprocess.run(
|
||||
[
|
||||
"gh", "issue", "list",
|
||||
"--repo", REPO,
|
||||
"--search", title_prefix,
|
||||
"--search", f'"{ISSUE_TITLE}" in:title',
|
||||
"--state", "open",
|
||||
"--json", "title",
|
||||
"--json", "number,title,body",
|
||||
"--limit", "10",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**os.environ, "GH_TOKEN": GH_TOKEN},
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return False
|
||||
print(f" ⚠️ gh issue list failed: {result.stderr.strip()}", file=sys.stderr)
|
||||
return None
|
||||
issues = json.loads(result.stdout or "[]")
|
||||
return any(title_prefix.lower() in i["title"].lower() for i in issues)
|
||||
for issue in issues:
|
||||
if issue["title"] == ISSUE_TITLE:
|
||||
return issue
|
||||
return None
|
||||
|
||||
|
||||
def create_issue(review: dict):
|
||||
stars = "⭐" * review["rating"]
|
||||
title = f"[Play Store Review] {review['rating']}★ from {review['author']} on {review['date']}"
|
||||
body = f"""**Rating**: {stars} ({review['rating']}/5)
|
||||
**Author**: {review['author']}
|
||||
**Date**: {review['date']}
|
||||
**Language**: {review['lang']}
|
||||
**Review ID**: `{review['review_id']}`
|
||||
def build_body(reviews, review_ids_marker):
|
||||
total = len(reviews)
|
||||
avg = sum(r["rating"] for r in reviews) / total if total else 0
|
||||
by_rating = Counter(r["rating"] for r in reviews)
|
||||
terms = common_terms(reviews)
|
||||
|
||||
---
|
||||
lines = [
|
||||
f"<!-- review-triage:{review_ids_marker} -->",
|
||||
"",
|
||||
f"## Play Store review triage — last {DAYS_BACK} days",
|
||||
"",
|
||||
f"**Reviews in window:** {total} | **Average rating:** {avg:.1f}★",
|
||||
"",
|
||||
"### Rating breakdown",
|
||||
"",
|
||||
"| Stars | Count |",
|
||||
"| --- | --- |",
|
||||
]
|
||||
for stars in (1, 2, 3):
|
||||
lines.append(f"| {stars}★ | {by_rating.get(stars, 0)} |")
|
||||
|
||||
> {review['text']}
|
||||
lines += [
|
||||
"",
|
||||
"### Common terms (word-frequency summary, not verbatim quotes)",
|
||||
"",
|
||||
(", ".join(f"`{t}`" for t in terms) if terms else "_not enough signal_"),
|
||||
"",
|
||||
"### Review references",
|
||||
"",
|
||||
"Look these up in Play Console → Reviews by ID for full context. "
|
||||
"No reviewer name or review text is reproduced here.",
|
||||
"",
|
||||
"| Review ID | Rating | Date | Language |",
|
||||
"| --- | --- | --- | --- |",
|
||||
]
|
||||
for r in sorted(reviews, key=lambda r: (r["rating"], r["date"])):
|
||||
lines.append(f"| `{r['review_id']}` | {r['rating']}★ | {r['date']} | {r['lang']} |")
|
||||
|
||||
---
|
||||
lines += [
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
"*This issue is automatically maintained by the "
|
||||
"[triage-reviews workflow](/.github/workflows/triage-reviews.yml). "
|
||||
"It is updated in place as new low-rated reviews appear in the window "
|
||||
"and intentionally contains no author names or raw review text.*",
|
||||
]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
*This issue was automatically created by the [triage-reviews workflow](/.github/workflows/triage-reviews.yml).
|
||||
Reply to the user in Play Console if applicable.*
|
||||
|
||||
**Labels**: `user-feedback`, `play-store-review`
|
||||
"""
|
||||
subprocess.run(
|
||||
[
|
||||
"gh", "issue", "create",
|
||||
"--repo", REPO,
|
||||
"--title", title,
|
||||
"--body", body,
|
||||
"--label", "user-feedback",
|
||||
],
|
||||
check=True,
|
||||
env={**os.environ, "GH_TOKEN": GH_TOKEN},
|
||||
)
|
||||
print(f" ✅ Created issue: {title}")
|
||||
def write_issue(existing, body):
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False) as f:
|
||||
f.write(body)
|
||||
body_path = f.name
|
||||
try:
|
||||
if existing:
|
||||
subprocess.run(
|
||||
[
|
||||
"gh", "issue", "edit", str(existing["number"]),
|
||||
"--repo", REPO,
|
||||
"--body-file", body_path,
|
||||
],
|
||||
check=True,
|
||||
env={**os.environ, "GH_TOKEN": GH_TOKEN},
|
||||
)
|
||||
print(f" ✅ Updated issue #{existing['number']}")
|
||||
else:
|
||||
subprocess.run(
|
||||
[
|
||||
"gh", "issue", "create",
|
||||
"--repo", REPO,
|
||||
"--title", ISSUE_TITLE,
|
||||
"--body-file", body_path,
|
||||
"--label", "user-feedback",
|
||||
],
|
||||
check=True,
|
||||
env={**os.environ, "GH_TOKEN": GH_TOKEN},
|
||||
)
|
||||
print(" ✅ Created triage issue")
|
||||
finally:
|
||||
os.unlink(body_path)
|
||||
|
||||
|
||||
def main():
|
||||
print(f"Fetching Play Store reviews for {PACKAGE_NAME} (last {DAYS_BACK} days)...")
|
||||
service = get_service()
|
||||
service = env_client()
|
||||
reviews = fetch_reviews(service)
|
||||
print(f"Found {len(reviews)} review(s) in window.")
|
||||
|
||||
# Triage: create issues for 1-3 star reviews (potential bugs/problems)
|
||||
actionable = [r for r in reviews if r["rating"] <= 3]
|
||||
# Material = actionable = potential bugs/problems worth triaging.
|
||||
actionable = sorted({r["review_id"] for r in reviews if r["rating"] <= 3 and r["review_id"]})
|
||||
actionable_reviews = [r for r in reviews if r["review_id"] in set(actionable)]
|
||||
current_marker = ",".join(actionable)
|
||||
print(f"Actionable (≤3★): {len(actionable)}")
|
||||
|
||||
for review in actionable:
|
||||
title_prefix = f"Play Store Review] {review['rating']}★ from {review['author']} on {review['date']}"
|
||||
if issue_exists(title_prefix):
|
||||
print(f" ⏭ Already exists: {review['author']} {review['date']}")
|
||||
continue
|
||||
create_issue(review)
|
||||
time.sleep(1) # avoid GitHub API rate limit
|
||||
if not actionable:
|
||||
print("No actionable reviews in window. Skipping.")
|
||||
return
|
||||
|
||||
existing = find_existing_issue()
|
||||
old_marker = ""
|
||||
if existing:
|
||||
match = MARKER_RE.search(existing.get("body") or "")
|
||||
if match:
|
||||
old_marker = match.group(1)
|
||||
|
||||
if current_marker == old_marker:
|
||||
print(" ⏭ No change in actionable review set since last run. Skipping.")
|
||||
return
|
||||
|
||||
body = build_body(actionable_reviews, current_marker)
|
||||
write_issue(existing, body)
|
||||
|
||||
# Summary
|
||||
avg = sum(r["rating"] for r in reviews) / len(reviews) if reviews else 0
|
||||
print(f"\nSummary: {len(reviews)} reviews, avg rating {avg:.1f}★, {len(actionable)} issues created/checked.")
|
||||
print(f"\nSummary: {len(reviews)} reviews, avg rating {avg:.1f}★, {len(actionable)} actionable.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in New Issue
Block a user