molecule-ai/molecule-core
51
0
Fork 2
Code Issues 164 Pull Requests 154 Actions 14 Packages Projects Releases Wiki Activity

Canary failing: staging SaaS smoke #1646

New Issue
Closed
opened 2026-05-21 19:16:39 +00:00 by gitea-actions · 22 comments
gitea-actions bot commented 2026-05-21 19:16:39 +00:00
Copy Link
Copy Source

Smoke run failed at 2026-05-21T19:16:39Z.

Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91241

This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.

Smoke run failed at 2026-05-21T19:16:39Z. Run: https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91241 This issue auto-closes on the next green smoke run. Consecutive failures add a comment here rather than a new issue.
gitea-actions bot commented 2026-05-21 19:34:42 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91288

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91288
gitea-actions bot commented 2026-05-21 20:04:46 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91419

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91419
gitea-actions bot commented 2026-05-21 20:32:28 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91521

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91521
gitea-actions bot commented 2026-05-21 21:06:14 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91624

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91624
gitea-actions bot commented 2026-05-21 21:34:29 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91676

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91676
gitea-actions bot commented 2026-05-21 22:04:27 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91758

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91758
gitea-actions bot commented 2026-05-21 22:38:51 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91801

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91801
gitea-actions bot commented 2026-05-21 23:08:38 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91855

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91855
gitea-actions bot commented 2026-05-21 23:36:31 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91929

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91929
gitea-actions bot commented 2026-05-22 00:08:12 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91968

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/91968
gitea-actions bot commented 2026-05-22 00:42:48 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92158

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92158
gitea-actions bot commented 2026-05-22 01:12:54 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92283

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92283
gitea-actions bot commented 2026-05-22 01:39:28 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92374

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92374
gitea-actions bot commented 2026-05-22 02:04:35 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92421

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92421
gitea-actions bot commented 2026-05-22 02:34:45 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92444

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92444
gitea-actions bot commented 2026-05-22 03:07:47 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92530

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92530
gitea-actions bot commented 2026-05-22 03:34:33 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92552

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92552
gitea-actions bot commented 2026-05-22 04:08:57 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92593

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92593
gitea-actions bot commented 2026-05-22 04:34:32 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92659

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92659
gitea-actions bot commented 2026-05-22 05:06:25 +00:00
Copy Link
Copy Source

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92780

Smoke still failing. https://git.moleculesai.app/molecule-ai/molecule-core/actions/runs/92780
gitea-actions bot commented 2026-05-22 05:34:42 +00:00
Copy Link
Copy Source

Smoke recovered at 2026-05-22T05:34:42Z. Closing.

Smoke recovered at 2026-05-22T05:34:42Z. Closing.
gitea-actions bot closed this issue 2026-05-22 05:34:43 +00:00
cp-be commented 2026-05-22 16:37:45 +00:00
Member
Copy Link
Copy Source

Diagnosis (assistant 091a9180, pulled actual canary logs from Gitea storage)

This is a flaky provisioning-latency issue, not a regression-introduced bisectable failure.

Last 15 staging SaaS canary runs alternate success/fail without a clean green→red transition:

Run SHA Status
93131 01087ddb ✅ success
93032 7fb0da3e ❌ FAIL
93007 805486e3 ✅ success
92971 8b8b3320 ✅ success
92936 bad66993 ✅ success
92916 8c3234e4 ✅ success
92896 741bb110 ❌ FAIL
92868 3a82e1f1 ✅ success
92842 f7183cc0 ❌ FAIL
92819 680434a8 ✅ success
92706 65f4ffb0 ❌ FAIL
92667 6f98ac06 ✅ success

Most recent failure (run 93032 / task 145341) logs show the canary makes it through:

  • CP healthy ✓
  • Tenant org created ✓
  • Tenant provisioning completes (running) ✓
  • Per-tenant admin token retrieved ✓
  • TLS / DNS propagation ✓
  • Parent + child workspace POSTs accepted ✓

Then hits step 7/11 Waiting for workspace(s) to reach status=online (up to 30 min — hermes cold boot) with PARENT at status=provisioning — and the job times out before workspace transitions to online. Log truncates at the 30-min timeout boundary.

Root cause class: provisioning latency variance. Some runs the workspace EC2 + container boot finishes within 30 min, some it doesn't. The 30-min timeout is the script's MAX (E2E_PROVISION_TIMEOUT_SECS default — see test_staging_full_saas.sh 7/11 step). When tenant provisioning takes >30 min, the canary fails — but the workspace itself eventually does come online (just past the timeout).

Not a code regression. The same SHA could pass and fail on re-run.

Recommended next step

Not a bisect target. Two paths:

  1. Quick mitigation: raise the 30-min timeout to e.g. 45 min — eats more CI time but eliminates the flap until root-caused. Cheap, reversible.
  2. Real fix: investigate provisioning latency distribution. Pull provisioned_at - status=running from org_instances for the last N tenant orgs, plot p50/p95/p99. If p95 is approaching 30 min, the canary is structurally too tight. Likely culprits: cold AMI pull, cloudflared tunnel registration race, secrets.d hydration. RFC-worthy.

Engineer-B's bisect approach won't find a culprit (no clean transition) — recommend rescoping its task to (a) compute the latency histogram from org_instances, (b) propose timeout adjustment OR (c) identify the slowest provisioning step.

Generated with Claude Code

## Diagnosis (assistant 091a9180, pulled actual canary logs from Gitea storage) **This is a flaky provisioning-latency issue, not a regression-introduced bisectable failure.** Last 15 staging SaaS canary runs alternate success/fail without a clean green→red transition: | Run | SHA | Status | |---|---|---| | 93131 | 01087ddb | ✅ success | | 93032 | 7fb0da3e | ❌ FAIL | | 93007 | 805486e3 | ✅ success | | 92971 | 8b8b3320 | ✅ success | | 92936 | bad66993 | ✅ success | | 92916 | 8c3234e4 | ✅ success | | 92896 | 741bb110 | ❌ FAIL | | 92868 | 3a82e1f1 | ✅ success | | 92842 | f7183cc0 | ❌ FAIL | | 92819 | 680434a8 | ✅ success | | 92706 | 65f4ffb0 | ❌ FAIL | | 92667 | 6f98ac06 | ✅ success | Most recent failure (run 93032 / task 145341) logs show the canary makes it through: - CP healthy ✓ - Tenant org created ✓ - Tenant provisioning completes (running) ✓ - Per-tenant admin token retrieved ✓ - TLS / DNS propagation ✓ - Parent + child workspace POSTs accepted ✓ Then hits step `7/11 Waiting for workspace(s) to reach status=online (up to 30 min — hermes cold boot)` with PARENT at `status=provisioning` — and the job times out before workspace transitions to `online`. Log truncates at the 30-min timeout boundary. **Root cause class: provisioning latency variance.** Some runs the workspace EC2 + container boot finishes within 30 min, some it doesn't. The 30-min timeout is the script's MAX (`E2E_PROVISION_TIMEOUT_SECS` default — see test_staging_full_saas.sh `7/11` step). When tenant provisioning takes >30 min, the canary fails — but the workspace itself eventually does come online (just past the timeout). **Not a code regression.** The same SHA could pass and fail on re-run. ## Recommended next step Not a bisect target. Two paths: 1. **Quick mitigation**: raise the 30-min timeout to e.g. 45 min — eats more CI time but eliminates the flap until root-caused. Cheap, reversible. 2. **Real fix**: investigate provisioning latency distribution. Pull `provisioned_at - status=running` from `org_instances` for the last N tenant orgs, plot p50/p95/p99. If p95 is approaching 30 min, the canary is structurally too tight. Likely culprits: cold AMI pull, cloudflared tunnel registration race, secrets.d hydration. RFC-worthy. Engineer-B's bisect approach won't find a culprit (no clean transition) — recommend rescoping its task to (a) compute the latency histogram from `org_instances`, (b) propose timeout adjustment OR (c) identify the slowest provisioning step. Generated with Claude Code
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1646
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?