molecule-ai/molecule-core
52
0
Fork 2
Code Issues 175 Pull Requests 20 Actions 5 Packages Projects Releases Wiki Activity

[main-red] molecule-ai/molecule-core: 8c00c8add7 #2886

New Issue
Closed
opened 2026-06-14 23:07:13 +00:00 by gitea-actions · 3 comments
gitea-actions bot commented 2026-06-14 23:07:13 +00:00
Copy Link
Copy Source

Main is RED on molecule-ai/molecule-core at 8c00c8add7

Commit: https://git.moleculesai.app/molecule-ai/molecule-core/commit/8c00c8add797d80a76befc17b00af5fd0db0385a

Auto-filed by .gitea/workflows/main-red-watchdog.yml (Option C of the main-never-red directive). Per feedback_no_such_thing_as_flakes + feedback_fix_root_not_symptom: investigate the root cause; do NOT revert as a reflex. The watchdog itself never reverts.

Failed status contexts

  • E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push) — failure → logs
    • Failing after 5m59s
  • E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push) — failure → logs
    • Failing after 8m30s
  • E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push) — failure → logs
    • Failing after 17m54s

Resolution path

  1. Read the failed logs (links above).
  2. If reproducible locally, fix forward in a PR targeting main.
  3. If the failure is a real flake — STOP. Per feedback_no_such_thing_as_flakes, intermittent failures are real bugs. Investigate to root cause; do not mark as flake.
  4. If the failure is blocking unrelated work for >1 hour, file a follow-up issue and assign someone. Do NOT revert without a human GO per feedback_prod_apply_needs_hongming_chat_go (branch protection is a prod surface).

Debug

{
  "all_contexts": [
    {
      "context": "E2E Chat / detect-changes (push)",
      "state": "success"
    },
    {
      "context": "E2E API Smoke Test / detect-changes (push)",
      "state": "success"
    },
    {
      "context": "CI / Detect changes (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge (compile+skip) (push)",
      "state": "success"
    },
    {
      "context": "CI / Canvas (Next.js) (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging Canvas (Playwright) / detect-changes (push)",
      "state": "success"
    },
    {
      "context": "lint-no-coe-on-required / lint-no-coe-on-required (push)",
      "state": "success"
    },
    {
      "context": "CI / Canvas Deploy Status (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / pr-validate (push)",
      "state": "success"
    },
    {
      "context": "CI / Shellcheck (E2E scripts) (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging Canvas (Playwright) / Canvas tabs E2E (push)",
      "state": "success"
    },
    {
      "context": "E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push)",
      "state": "success"
    },
    {
      "context": "Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (stub) (push)",
      "state": "success"
    },
    {
      "context": "Handlers Postgres Integration / Handlers Postgres Integration (push)",
      "state": "success"
    },
    {
      "context": "Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (real image + MiniMax LLM, advisory) (push)",
      "state": "success"
    },
    {
      "context": "Harness Replays / Harness Replays (push)",
      "state": "success"
    },
    {
      "context": "E2E Chat / E2E Chat (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Workspace Requests (core#2606) (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Platform Agent (push)",
      "state": "success"
    },
    {
      "context": "E2E API Smoke Test / E2E API Smoke Test (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge user_tasks (push)",
      "state": "success"
    },
    {
      "context": "publish-workspace-server-image / build-and-push (push)",
      "state": "success"
    },
    {
      "context": "CI / Platform (Go) (push)",
      "state": "success"
    },
    {
      "context": "CI / all-required (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging External Runtime / E2E Staging External Runtime (push)",
      "state": "success"
    },
    {
      "context": "E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)",
      "state": "failure"
    },
    {
      "context": "publish-workspace-server-image / Production auto-deploy (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)",
      "state": "failure"
    },
    {
      "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)",
      "state": "failure"
    }
  ],
  "branch": "main",
  "combined_state": "failure",
  "failed_contexts": [
    "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)",
    "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)",
    "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)"
  ],
  "recheck_combined_state": "failure",
  "recheck_failed_contexts": [
    "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)",
    "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)",
    "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)"
  ],
  "sha": "8c00c8add797d80a76befc17b00af5fd0db0385a"
}

This issue is idempotent: the watchdog runs hourly at :05 and edits this body in place. When main returns to green, the watchdog will close this issue automatically with a "main returned to green" comment.

# Main is RED on `molecule-ai/molecule-core` at `8c00c8add7` Commit: <https://git.moleculesai.app/molecule-ai/molecule-core/commit/8c00c8add797d80a76befc17b00af5fd0db0385a> Auto-filed by `.gitea/workflows/main-red-watchdog.yml` (Option C of the [main-never-red directive](https://git.moleculesai.app/molecule-ai/molecule-core/issues/420)). Per `feedback_no_such_thing_as_flakes` + `feedback_fix_root_not_symptom`: investigate the root cause; do NOT revert as a reflex. The watchdog itself never reverts. ## Failed status contexts - **E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/367546/jobs/503216) - Failing after 5m59s - **E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/367546/jobs/503215) - Failing after 8m30s - **E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/367546/jobs/503219) - Failing after 17m54s ## Resolution path 1. Read the failed logs (links above). 2. If reproducible locally, fix forward in a PR targeting `main`. 3. If the failure is a real flake — STOP. Per `feedback_no_such_thing_as_flakes`, intermittent failures are real bugs. Investigate to root cause; do not mark as flake. 4. If the failure is blocking unrelated work for >1 hour, file a follow-up issue and assign someone. Do NOT revert without a human GO per `feedback_prod_apply_needs_hongming_chat_go` (branch protection is a prod surface). ## Debug ```json { "all_contexts": [ { "context": "E2E Chat / detect-changes (push)", "state": "success" }, { "context": "E2E API Smoke Test / detect-changes (push)", "state": "success" }, { "context": "CI / Detect changes (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge (compile+skip) (push)", "state": "success" }, { "context": "CI / Canvas (Next.js) (push)", "state": "success" }, { "context": "E2E Staging Canvas (Playwright) / detect-changes (push)", "state": "success" }, { "context": "lint-no-coe-on-required / lint-no-coe-on-required (push)", "state": "success" }, { "context": "CI / Canvas Deploy Status (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / pr-validate (push)", "state": "success" }, { "context": "CI / Shellcheck (E2E scripts) (push)", "state": "success" }, { "context": "E2E Staging Canvas (Playwright) / Canvas tabs E2E (push)", "state": "success" }, { "context": "E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (local) (push)", "state": "success" }, { "context": "Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (stub) (push)", "state": "success" }, { "context": "Handlers Postgres Integration / Handlers Postgres Integration (push)", "state": "success" }, { "context": "Local Provision Lifecycle E2E / Local Provision Lifecycle E2E (real image + MiniMax LLM, advisory) (push)", "state": "success" }, { "context": "Harness Replays / Harness Replays (push)", "state": "success" }, { "context": "E2E Chat / E2E Chat (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Workspace Requests (core#2606) (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Platform Agent (push)", "state": "success" }, { "context": "E2E API Smoke Test / E2E API Smoke Test (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge user_tasks (push)", "state": "success" }, { "context": "publish-workspace-server-image / build-and-push (push)", "state": "success" }, { "context": "CI / Platform (Go) (push)", "state": "success" }, { "context": "CI / all-required (push)", "state": "success" }, { "context": "E2E Staging External Runtime / E2E Staging External Runtime (push)", "state": "success" }, { "context": "E2E Peer Visibility (literal MCP list_peers) / E2E Peer Visibility (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)", "state": "failure" }, { "context": "publish-workspace-server-image / Production auto-deploy (push)", "state": "success" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)", "state": "failure" }, { "context": "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)", "state": "failure" } ], "branch": "main", "combined_state": "failure", "failed_contexts": [ "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)", "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)", "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)" ], "recheck_combined_state": "failure", "recheck_failed_contexts": [ "E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push)", "E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push)", "E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push)" ], "sha": "8c00c8add797d80a76befc17b00af5fd0db0385a" } ``` _This issue is idempotent: the watchdog runs hourly at `:05` and edits this body in place. When `main` returns to green, the watchdog will close this issue automatically with a "main returned to green" comment._
agent-dev-a commented 2026-06-15 01:11:42 +00:00
Member
Copy Link
Copy Source

Root-cause read of the three failing runs (367546 / jobs 503215, 503216, 503219) on head 8c00c8add7:

  • E2E Staging SaaS and E2E Staging Platform Boot fail at the Verify admin token present step with CP_STAGING_ADMIN_API_TOKEN secret not set.
  • They also report AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY not set.
  • The E2E Staging Concierge Creates Workspace job additionally hits A2A parent failed after 1 attempt(s) (curl_rc=28, http=000) — consistent with the parent failing to boot because the same staging admin token + AWS creds are absent.
  • The CP staging health preflight also reports Staging CP unhealthy, which is an infra signal, not a workspace code signal.

These failures are environment/secret configuration, not a regression in 8c00c8add7 itself. The workflow (.gitea/workflows/e2e-staging-saas.yml) requires repo/org secrets that are currently missing or expired. Per the issue guidance, this should NOT be reverted; it needs the missing CP_STAGING_ADMIN_API_TOKEN, AWS leak-check credentials, and a healthy staging CP to be restored. Recommending SRE/ops review of the secret store.

Root-cause read of the three failing runs (367546 / jobs 503215, 503216, 503219) on head `8c00c8add7`: - `E2E Staging SaaS` and `E2E Staging Platform Boot` fail at the **Verify admin token present** step with `CP_STAGING_ADMIN_API_TOKEN secret not set`. - They also report `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` not set. - The `E2E Staging Concierge Creates Workspace` job additionally hits `A2A parent failed after 1 attempt(s) (curl_rc=28, http=000)` — consistent with the parent failing to boot because the same staging admin token + AWS creds are absent. - The CP staging health preflight also reports `Staging CP unhealthy`, which is an infra signal, not a workspace code signal. These failures are **environment/secret configuration**, not a regression in `8c00c8add7` itself. The workflow (`.gitea/workflows/e2e-staging-saas.yml`) requires repo/org secrets that are currently missing or expired. Per the issue guidance, this should NOT be reverted; it needs the missing `CP_STAGING_ADMIN_API_TOKEN`, AWS leak-check credentials, and a healthy staging CP to be restored. Recommending SRE/ops review of the secret store.
gitea-actions bot commented 2026-06-15 02:07:22 +00:00
Copy Link
Copy Source

The failing contexts from this SHA (8c00c8add7) have recovered on current HEAD 6a2e128152: E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push), E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push), E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push). Main is still red for other reasons; see the current [main-red] issue for 6a2e128152.

The failing contexts from this SHA (`8c00c8add7`) have recovered on current HEAD `6a2e128152`: E2E Staging SaaS (full lifecycle) / E2E Staging Platform Boot (push), E2E Staging SaaS (full lifecycle) / E2E Staging SaaS (push), E2E Staging SaaS (full lifecycle) / E2E Staging Concierge Creates Workspace (push). Main is still red for other reasons; see the current `[main-red]` issue for `6a2e128152`.
gitea-actions bot closed this issue 2026-06-15 02:07:23 +00:00
agent-dev-a commented 2026-06-15 02:28:15 +00:00
Member
Copy Link
Copy Source

Closing — current main (6a2e1281) is healthy. The staging E2E contexts that failed on 8c00c8add7 are no longer red; the only remaining red context is the advisory Local Provision Lifecycle E2E (real image + MiniMax LLM), which is a separate non-gating/flaky lane tracked elsewhere. The original root cause was missing staging secrets/infra, not a code regression.

Closing — current `main` (6a2e1281) is healthy. The staging E2E contexts that failed on 8c00c8add7 are no longer red; the only remaining red context is the advisory `Local Provision Lifecycle E2E (real image + MiniMax LLM)`, which is a separate non-gating/flaky lane tracked elsewhere. The original root cause was missing staging secrets/infra, not a code regression.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2886
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?