forked from molecule-ai/molecule-core
6e0eb2ddc9
Recurring failure pattern in redeploy-tenants-on-staging: ##[error]redeploy-fleet returned HTTP 500 ##[error]Process completed with exit code 1. with the per-tenant breakdown in the response body showing the failures were on ephemeral e2e-* tenants (saas/canvas/ext) whose parent E2E run torn them down mid-redeploy — SSM exit=2 because the EC2 was already terminating, or healthz timeout because the CF tunnel was already gone. The actual operator-facing tenants (dryrun-98407, demo-prep, etc) all rolled fine in the same call. This shape repeats every staging push that overlaps an active E2E run. The downstream `Verify each staging tenant /buildinfo matches published SHA` step ALREADY distinguishes STALE vs UNREACHABLE for exactly this reason (per #2402); only the top-level `if HTTP_CODE != 200; exit 1` gate misclassifies the race. Filter: HTTP 500 + every failed slug matches `^e2e-` → soft-warn and fall through to verify. Any non-e2e-* failure or non-500 HTTP remains a hard fail, with the failed non-e2e slugs surfaced in the error so the operator doesn't have to dig the response body out of CI. Verified the gate logic with 6 synthetic CP responses (happy / e2e-only race / mixed real+e2e fail / non-200 / 200+ok=false / all-real-fail) — all behave correctly. prod's redeploy-tenants-on-main is intentionally NOT touched: prod CP serves no e2e-* tenants, so the race can't occur there and the strict gate is the right behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| scripts | ||
| workflows | ||
| CODEOWNERS | ||
| dependabot.yml | ||