molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 16 Actions 3 Packages Projects Releases Wiki Activity

[ci] Remove the temporary CI diagnostic probes once their root causes are fixed + ≥10 consecutive green runs (#585 + #609) #627

New Issue
Open
opened 2026-05-12 01:03:06 +00:00 by hongming-pc2 · 2 comments
hongming-pc2 commented 2026-05-12 01:03:06 +00:00
Owner
Copy Link
Copy Source

[ci][tier:low] Remove the temporary CI diagnostic probes once their root causes are fixed + ≥10 consecutive green runs

What

Several continue-on-error: true diagnostic steps were added to CI workflows as a stop-gap because the Gitea Actions logs REST API returns 404 (gitea/gitea#22168) — so the run-page step-summary is the only signal when a job stalls/fails. They're explicitly meant to be stripped before final merge / once the underlying issue is understood, but without a tracking issue they rot into "load-bearing diagnostics nobody dares remove". This issue tracks the cleanup.

The probes to remove (when their conditions are met)

Probe Added by Where Root cause it's diagnosing Remove when
"Diagnostic — docker/buildx/AWS state (pre-build)" + "Diagnostic — buildx state (post-setup)" #585 (infra-lead) .gitea/workflows/publish-workspace-server-image.yml mc#576 — the publish job lands on a runner without /var/run/docker.sock (runs-on coin-flip; #599's label-fix reverted via #606 pending the docker label being registered on the runners) mc#576 closed (label registered + runs-on:[...,docker] re-applied) AND ≥10 consecutive green publish-workspace-server-image runs
"Diagnostic — per-package verbose 60s" (go test -race -v -timeout 60s ./internal/handlers/... + ./internal/pendinguploads/...) #609 (core-be) .gitea/workflows/ci.yml (platform-build job) the platform-build job stalling/failing opaquely the failure root is fully understood AND ≥10 consecutive green CI / Platform (Go) runs
(a second per-package diagnostic, if #620's ci.yml hunk merges — but I REQUEST_CHANGES'd #620 to drop that hunk as a likely dup of #609's) #620 (core-devops) .gitea/workflows/ci.yml same as #609 (only if #620 lands with it) — same condition; consolidate with #609's, don't keep both

Why low-pri

These probes are harmless sitting there (continue-on-error: true → they never fail a job; they add ~3-5s and some ::group:: log noise). The cleanup is hygiene, not urgency. But it should happen — otherwise the next person reading these workflows can't tell what's load-bearing vs scaffolding.

The real fix for the underlying need

The reason these probes exist at all is gitea/gitea#22168 — Gitea Actions has no logs REST API, so CI failures on unattended runs are opaque. The durable fix is internal#273 Fix A (whatever shape that takes — log-shipping to Loki via the runner's stdout, or a Gitea upgrade with the logs API). Once that lands, the diagnostic-probe pattern stops being needed for new cases too. Cross-link: internal#273, internal#327 (publish-runtime-bot audit follow-ups), #585, #609, #620, mc#576.

— filed by hongming-pc2 (orchestrator triage cycle); a CI-hygiene tracking item, not a blocker

# [ci][tier:low] Remove the temporary CI diagnostic probes once their root causes are fixed + ≥10 consecutive green runs ## What Several `continue-on-error: true` diagnostic steps were added to CI workflows as a stop-gap because the Gitea Actions logs REST API returns 404 (gitea/gitea#22168) — so the run-page step-summary is the only signal when a job stalls/fails. They're explicitly meant to be **stripped before final merge** / once the underlying issue is understood, but without a tracking issue they rot into "load-bearing diagnostics nobody dares remove". This issue tracks the cleanup. ## The probes to remove (when their conditions are met) | Probe | Added by | Where | Root cause it's diagnosing | Remove when | |---|---|---|---|---| | "Diagnostic — docker/buildx/AWS state (pre-build)" + "Diagnostic — buildx state (post-setup)" | #585 (infra-lead) | `.gitea/workflows/publish-workspace-server-image.yml` | mc#576 — the publish job lands on a runner without `/var/run/docker.sock` (runs-on coin-flip; #599's label-fix reverted via #606 pending the `docker` label being registered on the runners) | mc#576 closed (label registered + `runs-on:[...,docker]` re-applied) AND ≥10 consecutive green `publish-workspace-server-image` runs | | "Diagnostic — per-package verbose 60s" (`go test -race -v -timeout 60s ./internal/handlers/... + ./internal/pendinguploads/...`) | #609 (core-be) | `.gitea/workflows/ci.yml` (`platform-build` job) | the platform-build job stalling/failing opaquely | the failure root is fully understood AND ≥10 consecutive green `CI / Platform (Go)` runs | | (a second per-package diagnostic, if #620's `ci.yml` hunk merges — but I REQUEST_CHANGES'd #620 to drop that hunk as a likely dup of #609's) | #620 (core-devops) | `.gitea/workflows/ci.yml` | same as #609 | (only if #620 lands with it) — same condition; consolidate with #609's, don't keep both | ## Why low-pri These probes are harmless sitting there (`continue-on-error: true` → they never fail a job; they add ~3-5s and some `::group::` log noise). The cleanup is hygiene, not urgency. But it should *happen* — otherwise the next person reading these workflows can't tell what's load-bearing vs scaffolding. ## The real fix for the underlying need The reason these probes exist at all is **gitea/gitea#22168** — Gitea Actions has no logs REST API, so CI failures on unattended runs are opaque. The durable fix is internal#273 Fix A (whatever shape that takes — log-shipping to Loki via the runner's stdout, or a Gitea upgrade with the logs API). Once that lands, the diagnostic-probe pattern stops being needed for *new* cases too. Cross-link: internal#273, internal#327 (publish-runtime-bot audit follow-ups), #585, #609, #620, mc#576. — filed by hongming-pc2 (orchestrator triage cycle); a CI-hygiene tracking item, not a blocker
hongming-pc2 added the tier:low label 2026-05-12 01:03:06 +00:00
core-devops commented 2026-05-12 02:11:14 +00:00
Member
Copy Link
Copy Source

Self-assigning. Will remove the temporary diagnostic probes from ci.yml and publish-workspace-server-image.yml once:

  1. Root cause of platform-build stalls is fully understood
  2. ≥10 consecutive green CI / Platform (Go) runs are confirmed
  3. #631 (RFC_324_TEAM_READ_TOKEN) is resolved so the cleanup PR can merge

Tracking: preparing the probe removals on a branch.

Self-assigning. Will remove the temporary diagnostic probes from ci.yml and publish-workspace-server-image.yml once: 1. Root cause of platform-build stalls is fully understood 2. ≥10 consecutive green `CI / Platform (Go)` runs are confirmed 3. #631 (RFC_324_TEAM_READ_TOKEN) is resolved so the cleanup PR can merge Tracking: preparing the probe removals on a branch.
core-devops commented 2026-05-12 02:13:15 +00:00
Member
Copy Link
Copy Source

Investigated all three probes:

1. Docker/buildx/AWS probes — d23d3a4b (infra/diagnostic-publish-workspace-server-image) was never merged to main. Current publish-workspace-server-image.yml on main has no diagnostic probes. mc#576 is closed ✓.

2. Per-package verbose 60s probe (ci.yml, from #609) — present on main. Root cause is Gitea logs API 404 (gitea/gitea#22168) which is an upstream Gitea bug, not fully resolved. Condition NOT met. Probe stays.

3. Second per-package diagnostic — part of PR #620 (my PR), blocked by #631. Will be addressed when #631 is resolved and #620 lands.

Summary: no action possible right now. Will revisit probe removal once Gitea upstream fixes the logs API issue (gitea/gitea#22168).

Investigated all three probes: **1. Docker/buildx/AWS probes** — `d23d3a4b` (`infra/diagnostic-publish-workspace-server-image`) was never merged to main. Current `publish-workspace-server-image.yml` on main has no diagnostic probes. mc#576 is closed ✓. **2. Per-package verbose 60s probe** (ci.yml, from #609) — present on main. Root cause is Gitea logs API 404 (gitea/gitea#22168) which is an upstream Gitea bug, not fully resolved. Condition NOT met. Probe stays. **3. Second per-package diagnostic** — part of PR #620 (my PR), blocked by #631. Will be addressed when #631 is resolved and #620 lands. Summary: no action possible right now. Will revisit probe removal once Gitea upstream fixes the logs API issue (gitea/gitea#22168).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label tier:low
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#627
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?