[main-red] molecule-ai/molecule-core: be3ac6dceb #897

New Issue

gitea-actions · 2026-05-13T22:10:37Z

gitea-actions commented

2026-05-13 22:10:37 +00:00

Main is RED on `molecule-ai/molecule-core` at `be3ac6dceb`

Commit: https://git.moleculesai.app/molecule-ai/molecule-core/commit/be3ac6dceb9fc0f5eadcaf482c4acc474a2eedbe

Auto-filed by .gitea/workflows/main-red-watchdog.yml (Option C of the main-never-red directive). Per feedback_no_such_thing_as_flakes + feedback_fix_root_not_symptom: investigate the root cause; do NOT revert as a reflex. The watchdog itself never reverts.

Failed status contexts

Runtime PR-Built Compatibility / PR-built wheel + import smoke (push) — failure → logs
- Failing after 12m9s
CI / Platform (Go) (push) — failure → logs
- Failing after 16m38s
CI / Platform (Go) (pull_request) — failure → logs
- Failing after 16m34s
publish-workspace-server-image / Production auto-deploy (push) — failure → logs
- Failing after 4m15s

Resolution path

Read the failed logs (links above).
If reproducible locally, fix forward in a PR targeting main.
If the failure is a real flake — STOP. Per feedback_no_such_thing_as_flakes, intermittent failures are real bugs. Investigate to root cause; do not mark as flake.
If the failure is blocking unrelated work for >1 hour, file a follow-up issue and assign someone. Do NOT revert without a human GO per feedback_prod_apply_needs_hongming_chat_go (branch protection is a prod surface).

Debug

{
  "all_contexts": [
    {
      "context": "sop-checklist-gate / gate (pull_request)",
      "state": "success"
    },
    {
      "context": "Ops Scripts Tests / Ops scripts (unittest) (push)",
      "state": "success"
    },
    {
      "context": "Harness Replays / Harness Replays (pull_request)",
      "state": "success"
    },
    {
      "context": "sop-tier-check / tier-check (pull_request)",
      "state": "success"
    },
    {
      "context": "Harness Replays / Harness Replays (push)",
      "state": "success"
    },
    {
      "context": "CI / Shellcheck (E2E scripts) (push)",
      "state": "success"
    },
    {
      "context": "CI / Canvas (Next.js) (push)",
      "state": "success"
    },
    {
      "context": "CI / Python Lint & Test (push)",
      "state": "success"
    },
    {
      "context": "Ops Scripts Tests / Ops scripts (unittest) (pull_request)",
      "state": "success"
    },
    {
      "context": "Handlers Postgres Integration / Handlers Postgres Integration (push)",
      "state": "success"
    },
    {
      "context": "E2E Staging Canvas (Playwright) / Canvas tabs E2E (push)",
      "state": "success"
    },
    {
      "context": "CI / Shellcheck (E2E scripts) (pull_request)",
      "state": "success"
    },
    {
      "context": "status-reaper / reap (push)",
      "state": "pending"
    },
    {
      "context": "E2E API Smoke Test / E2E API Smoke Test (push)",
      "state": "success"
    },
    {
      "context": "E2E API Smoke Test / E2E API Smoke Test (pull_request)",
      "state": "success"
    },
    {
      "context": "Continuous synthetic E2E (staging) / Synthetic E2E against staging (push)",
      "state": "pending"
    },
    {
      "context": "Handlers Postgres Integration / Handlers Postgres Integration (pull_request)",
      "state": "success"
    },
    {
      "context": "publish-workspace-server-image / build-and-push (push)",
      "state": "success"
    },
    {
      "context": "CI / Canvas Deploy Reminder (push)",
      "state": "success"
    },
    {
      "context": "CI / Python Lint & Test (pull_request)",
      "state": "success"
    },
    {
      "context": "Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request)",
      "state": "success"
    },
    {
      "context": "Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push)",
      "state": "success"
    },
    {
      "context": "Runtime PR-Built Compatibility / PR-built wheel + import smoke (push)",
      "state": "failure"
    },
    {
      "context": "CI / Canvas (Next.js) (pull_request)",
      "state": "success"
    },
    {
      "context": "CI / Platform (Go) (push)",
      "state": "failure"
    },
    {
      "context": "CI / Platform (Go) (pull_request)",
      "state": "failure"
    },
    {
      "context": "publish-workspace-server-image / Production auto-deploy (push)",
      "state": "failure"
    },
    {
      "context": "main-red-watchdog / watchdog (push)",
      "state": "pending"
    },
    {
      "context": "gitea-merge-queue / queue (push)",
      "state": "success"
    },
    {
      "context": "Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push)",
      "state": "success"
    }
  ],
  "branch": "main",
  "combined_state": "failure",
  "failed_contexts": [
    "Runtime PR-Built Compatibility / PR-built wheel + import smoke (push)",
    "CI / Platform (Go) (push)",
    "CI / Platform (Go) (pull_request)",
    "publish-workspace-server-image / Production auto-deploy (push)"
  ],
  "sha": "be3ac6dceb9fc0f5eadcaf482c4acc474a2eedbe"
}

This issue is idempotent: the watchdog runs hourly at :05 and edits this body in place. When main returns to green, the watchdog will close this issue automatically with a "main returned to green" comment.

# Main is RED on `molecule-ai/molecule-core` at `be3ac6dceb` Commit: <https://git.moleculesai.app/molecule-ai/molecule-core/commit/be3ac6dceb9fc0f5eadcaf482c4acc474a2eedbe> Auto-filed by `.gitea/workflows/main-red-watchdog.yml` (Option C of the [main-never-red directive](https://git.moleculesai.app/molecule-ai/molecule-core/issues/420)). Per `feedback_no_such_thing_as_flakes` + `feedback_fix_root_not_symptom`: investigate the root cause; do NOT revert as a reflex. The watchdog itself never reverts. ## Failed status contexts - **Runtime PR-Built Compatibility / PR-built wheel + import smoke (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/29588/jobs/1) - Failing after 12m9s - **CI / Platform (Go) (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/29563/jobs/1) - Failing after 16m38s - **CI / Platform (Go) (pull_request)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/29564/jobs/1) - Failing after 16m34s - **publish-workspace-server-image / Production auto-deploy (push)** — `failure` → [logs](/molecule-ai/molecule-core/actions/runs/29583/jobs/1) - Failing after 4m15s ## Resolution path 1. Read the failed logs (links above). 2. If reproducible locally, fix forward in a PR targeting `main`. 3. If the failure is a real flake — STOP. Per `feedback_no_such_thing_as_flakes`, intermittent failures are real bugs. Investigate to root cause; do not mark as flake. 4. If the failure is blocking unrelated work for >1 hour, file a follow-up issue and assign someone. Do NOT revert without a human GO per `feedback_prod_apply_needs_hongming_chat_go` (branch protection is a prod surface). ## Debug ```json { "all_contexts": [ { "context": "sop-checklist-gate / gate (pull_request)", "state": "success" }, { "context": "Ops Scripts Tests / Ops scripts (unittest) (push)", "state": "success" }, { "context": "Harness Replays / Harness Replays (pull_request)", "state": "success" }, { "context": "sop-tier-check / tier-check (pull_request)", "state": "success" }, { "context": "Harness Replays / Harness Replays (push)", "state": "success" }, { "context": "CI / Shellcheck (E2E scripts) (push)", "state": "success" }, { "context": "CI / Canvas (Next.js) (push)", "state": "success" }, { "context": "CI / Python Lint & Test (push)", "state": "success" }, { "context": "Ops Scripts Tests / Ops scripts (unittest) (pull_request)", "state": "success" }, { "context": "Handlers Postgres Integration / Handlers Postgres Integration (push)", "state": "success" }, { "context": "E2E Staging Canvas (Playwright) / Canvas tabs E2E (push)", "state": "success" }, { "context": "CI / Shellcheck (E2E scripts) (pull_request)", "state": "success" }, { "context": "status-reaper / reap (push)", "state": "pending" }, { "context": "E2E API Smoke Test / E2E API Smoke Test (push)", "state": "success" }, { "context": "E2E API Smoke Test / E2E API Smoke Test (pull_request)", "state": "success" }, { "context": "Continuous synthetic E2E (staging) / Synthetic E2E against staging (push)", "state": "pending" }, { "context": "Handlers Postgres Integration / Handlers Postgres Integration (pull_request)", "state": "success" }, { "context": "publish-workspace-server-image / build-and-push (push)", "state": "success" }, { "context": "CI / Canvas Deploy Reminder (push)", "state": "success" }, { "context": "CI / Python Lint & Test (pull_request)", "state": "success" }, { "context": "Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request)", "state": "success" }, { "context": "Sweep stale e2e-* orgs (staging) / Sweep e2e orgs (push)", "state": "success" }, { "context": "Runtime PR-Built Compatibility / PR-built wheel + import smoke (push)", "state": "failure" }, { "context": "CI / Canvas (Next.js) (pull_request)", "state": "success" }, { "context": "CI / Platform (Go) (push)", "state": "failure" }, { "context": "CI / Platform (Go) (pull_request)", "state": "failure" }, { "context": "publish-workspace-server-image / Production auto-deploy (push)", "state": "failure" }, { "context": "main-red-watchdog / watchdog (push)", "state": "pending" }, { "context": "gitea-merge-queue / queue (push)", "state": "success" }, { "context": "Staging SaaS smoke (every 30 min) / Staging SaaS smoke (push)", "state": "success" } ], "branch": "main", "combined_state": "failure", "failed_contexts": [ "Runtime PR-Built Compatibility / PR-built wheel + import smoke (push)", "CI / Platform (Go) (push)", "CI / Platform (Go) (pull_request)", "publish-workspace-server-image / Production auto-deploy (push)" ], "sha": "be3ac6dceb9fc0f5eadcaf482c4acc474a2eedbe" } ``` _This issue is idempotent: the watchdog runs hourly at `:05` and edits this body in place. When `main` returns to green, the watchdog will close this issue automatically with a "main returned to green" comment._

gitea-actions added the

tier:high

label 2026-05-13 22:10:38 +00:00

infra-sre commented

2026-05-13 22:19:53 +00:00

[infra-sre] Escalation: publish-workspace-server-image failing after 4m15s. Per mc#711 comment in the workflow, Docker daemon is not accessible on some ubuntu-latest runners (client-only in docker info). The diagnostic step (lines 68-82) is present but the workflow fails before completing. Likely cause: Docker daemon inaccessible on current runner + ECR auth step failing. Recommend: (1) check if runner hostname has accessible daemon, (2) verify AWS credentials are still valid for ECR push.

[infra-sre] Escalation: publish-workspace-server-image failing after 4m15s. Per mc#711 comment in the workflow, Docker daemon is not accessible on some ubuntu-latest runners (client-only in `docker info`). The diagnostic step (lines 68-82) is present but the workflow fails before completing. Likely cause: Docker daemon inaccessible on current runner + ECR auth step failing. Recommend: (1) check if runner hostname has accessible daemon, (2) verify AWS credentials are still valid for ECR push.

~~fullstack-engineer referenced this issue 2026-05-13 22:33:54 +00:00~~

fix: ListDelegations + 5 pre-existing test compilation errors #901

fullstack-engineer referenced this issue

2026-05-13 23:21:46 +00:00

fix: ListDelegations + 5 pre-existing test compilation errors #901

triage-operator closed this issue

2026-05-13 23:23:57 +00:00

Sign in to join this conversation.