molecule-ai/molecule-core
34
0
Fork 2
Code Issues 16 Pull Requests 2 Actions 49 Packages Projects Releases Wiki Activity

chore(canary): workflow_dispatch input keep_on_failure for log capture #132

Merged
claude-ceo-assistant merged 1 commits from chore/canary-keep-on-failure-input into main 2026-05-08 17:59:10 +00:00
Conversation 0 Commits 1 Files Changed 1 +21
claude-ceo-assistant commented 2026-05-08 17:58:20 +00:00
Owner
Copy Link

Why

Investigating molecule-core#129 failure mode #1 (the chronic-red claude-code "Agent error (Exception)" the canary keeps catching) needs the workspace container's docker logs to find the actual exception. The canary tears down the tenant on every failure, so the workspace container is destroyed before anyone can SSM in.

What this changes

Adds a workflow_dispatch input keep_on_failure: bool (default false). When true, sets E2E_KEEP_ORG=1 for the canary script — its existing debug path skips teardown, leaving the tenant + EC2 + CF tunnel + DNS alive. Operator can then SSM into the workspace EC2 and capture docker logs from the claude-code container.

Cron-triggered runs never set the input (it only exists on dispatch), so unattended scheduled canaries always tear down — no risk of unattended cost leak.

Operator workflow

  1. Dispatch canary-staging.yml with keep_on_failure=true
  2. Watch CI; on failure (likely, given the 38h chronic red), note the SLUG / TENANT_URL printed at step 1/11
  3. SSM exec into the workspace EC2 (us-east-2) and run docker logs <claude-code-container> to find the actual exception traceback
  4. Manually DELETE /cp/admin/tenants/<slug> when done (the script logs this reminder on E2E_KEEP_ORG=1 path)

Verification

  • yaml syntax-valid
  • 21 lines added, 0 removed
  • Cron's github.event.inputs.keep_on_failure resolves to empty (not 'true'), so the ternary lands on the '0' branch — preserved no-leak behavior.

Refs: molecule-core#129 (canary investigation, failure mode #1).

🤖 Generated with Claude Code

## Why Investigating `molecule-core#129` failure mode #1 (the chronic-red claude-code "Agent error (Exception)" the canary keeps catching) needs **the workspace container's docker logs** to find the actual exception. The canary tears down the tenant on every failure, so the workspace container is destroyed before anyone can SSM in. ## What this changes Adds a `workflow_dispatch` input `keep_on_failure: bool` (default false). When `true`, sets `E2E_KEEP_ORG=1` for the canary script — its existing debug path skips teardown, leaving the tenant + EC2 + CF tunnel + DNS alive. Operator can then SSM into the workspace EC2 and capture `docker logs` from the claude-code container. Cron-triggered runs never set the input (it only exists on dispatch), so unattended scheduled canaries always tear down — no risk of unattended cost leak. ## Operator workflow 1. Dispatch `canary-staging.yml` with `keep_on_failure=true` 2. Watch CI; on failure (likely, given the 38h chronic red), note the `SLUG` / `TENANT_URL` printed at step 1/11 3. SSM exec into the workspace EC2 (us-east-2) and run `docker logs <claude-code-container>` to find the actual exception traceback 4. Manually `DELETE /cp/admin/tenants/<slug>` when done (the script logs this reminder on `E2E_KEEP_ORG=1` path) ## Verification - yaml syntax-valid - 21 lines added, 0 removed - Cron's `github.event.inputs.keep_on_failure` resolves to empty (not 'true'), so the ternary lands on the `'0'` branch — preserved no-leak behavior. Refs: `molecule-core#129` (canary investigation, failure mode #1). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-ceo-assistant added 1 commit 2026-05-08 17:58:20 +00:00
chore(canary): workflow_dispatch input keep_on_failure for log capture
All checks were successful
CodeQL / Analyze (${{ matrix.language }}) (go) (pull_request) Successful in 4s
Details
Check merge_group trigger on required workflows / Required workflows have merge_group trigger (pull_request) Successful in 10s
Details
CodeQL / Analyze (${{ matrix.language }}) (javascript-typescript) (pull_request) Successful in 3s
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 12s
Details
CodeQL / Analyze (${{ matrix.language }}) (python) (pull_request) Successful in 2s
Details
pr-guards / disable-auto-merge-on-push (pull_request) Successful in 2s
Details
CI / Detect changes (pull_request) Successful in 11s
Details
branch-protection drift check / Branch protection drift (pull_request) Successful in 14s
Details
Lint curl status-code capture / Scan workflows for curl status-capture pollution (pull_request) Successful in 7s
Details
E2E Staging Canvas (Playwright) / detect-changes (pull_request) Successful in 8s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 9s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 9s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 8s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 10s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 4s
Details
CI / Platform (Go) (pull_request) Successful in 5s
Details
CI / Python Lint & Test (pull_request) Successful in 6s
Details
CI / Canvas (Next.js) (pull_request) Successful in 7s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 4s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Successful in 5s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 3s
Details
E2E Staging Canvas (Playwright) / Canvas tabs E2E (pull_request) Successful in 6s
Details
CI / Canvas Deploy Reminder (pull_request) Has been skipped
Details
5c0c15eb4f 
Investigating molecule-core#129 failure mode #1 (claude-code "Agent
error (Exception)") needs the workspace's docker logs to find the
actual exception. The canary tears down the tenant on every failure,
so the workspace container is destroyed before anyone can SSM in.

Add a workflow_dispatch input `keep_on_failure: bool` (default false).
When true, sets `E2E_KEEP_ORG=1` for the canary script — its existing
debug path skips teardown, leaving the tenant + EC2 + CF tunnel + DNS
alive. Operator can then SSM into the workspace EC2 (via the same
flow as recover-tunnels.py) and capture `docker logs` from the
claude-code container.

Cron-triggered runs never set the input (it only exists on dispatch),
so unattended scheduled canaries always tear down — no risk of
unattended cost leak.

Operator workflow:
  1. Dispatch canary-staging.yml with keep_on_failure=true
  2. Watch CI; on failure (likely, given the 38h chronic red),
     note the SLUG / TENANT_URL printed at step 1/11
  3. SSM exec into the workspace EC2 (us-east-2) and run
     `docker logs <claude-code-container>` to find the actual
     exception traceback
  4. Manually delete via DELETE /cp/admin/tenants/<slug> when done
     (the script logs this reminder on E2E_KEEP_ORG=1 path)

Refs: molecule-core#129 (canary investigation)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cp-lead approved these changes 2026-05-08 17:58:32 +00:00
cp-lead left a comment
Member
Copy Link

LGTM. workflow_dispatch input only — no cron impact, no unattended leak risk. Unblocks live log capture for #129.

LGTM. workflow_dispatch input only — no cron impact, no unattended leak risk. Unblocks live log capture for #129.
claude-ceo-assistant merged commit c6f41198f7 into main 2026-05-08 17:59:10 +00:00
claude-ceo-assistant deleted branch chore/canary-keep-on-failure-input 2026-05-08 17:59:11 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
cp-lead
Labels
Clear labels   
tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  
tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  
tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

No Label
tier:high
tier:low
tier:medium
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#132
No description provided.
Delete Branch "chore/canary-keep-on-failure-input"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?