forked from molecule-ai/molecule-core
5c0c15eb4f
Investigating molecule-core#129 failure mode #1 (claude-code "Agent error (Exception)") needs the workspace's docker logs to find the actual exception. The canary tears down the tenant on every failure, so the workspace container is destroyed before anyone can SSM in. Add a workflow_dispatch input `keep_on_failure: bool` (default false). When true, sets `E2E_KEEP_ORG=1` for the canary script — its existing debug path skips teardown, leaving the tenant + EC2 + CF tunnel + DNS alive. Operator can then SSM into the workspace EC2 (via the same flow as recover-tunnels.py) and capture `docker logs` from the claude-code container. Cron-triggered runs never set the input (it only exists on dispatch), so unattended scheduled canaries always tear down — no risk of unattended cost leak. Operator workflow: 1. Dispatch canary-staging.yml with keep_on_failure=true 2. Watch CI; on failure (likely, given the 38h chronic red), note the SLUG / TENANT_URL printed at step 1/11 3. SSM exec into the workspace EC2 (us-east-2) and run `docker logs <claude-code-container>` to find the actual exception traceback 4. Manually delete via DELETE /cp/admin/tenants/<slug> when done (the script logs this reminder on E2E_KEEP_ORG=1 path) Refs: molecule-core#129 (canary investigation) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| auto-tag-runtime.yml | ||
| block-internal-paths.yml | ||
| branch-protection-drift.yml | ||
| canary-staging.yml | ||
| canary-verify.yml | ||
| cascade-list-drift-gate.yml | ||
| check-merge-group-trigger.yml | ||
| check-migration-collisions.yml | ||
| ci.yml | ||
| codeql.yml | ||
| continuous-synth-e2e.yml | ||
| e2e-api.yml | ||
| e2e-staging-canvas.yml | ||
| e2e-staging-external.yml | ||
| e2e-staging-saas.yml | ||
| e2e-staging-sanity.yml | ||
| handlers-postgres-integration.yml | ||
| harness-replays.yml | ||
| lint-curl-status-capture.yml | ||
| pr-guards.yml | ||
| promote-latest.yml | ||
| publish-canvas-image.yml | ||
| publish-runtime.yml | ||
| publish-workspace-server-image.yml | ||
| railway-pin-audit.yml | ||
| redeploy-tenants-on-main.yml | ||
| redeploy-tenants-on-staging.yml | ||
| runtime-pin-compat.yml | ||
| runtime-prbuild-compat.yml | ||
| secret-pattern-drift.yml | ||
| secret-scan.yml | ||
| sweep-aws-secrets.yml | ||
| sweep-cf-orphans.yml | ||
| sweep-cf-tunnels.yml | ||
| sweep-stale-e2e-orgs.yml | ||
| test-ops-scripts.yml | ||