molecule-ai/molecule-core
52
0
Fork 2
Code Issues 184 Pull Requests 27 Actions 18 Packages Projects Releases Wiki Activity

Switch publish-image cache to type=registry (GHA cache emulator unreachable from buildkit) #1182

New Issue
Open
opened 2026-05-15 12:20:15 +00:00 by hongming · 0 comments
hongming commented 2026-05-15 12:20:15 +00:00
Owner
Copy Link
Copy Source

Problem

publish-image.yml in workspace templates was using cache-to: type=gha,mode=max + cache-from: type=gha for buildkit cache. The act_runner provides a GHA-emulator cache server (e.g. at 172.18.0.7:37871 on the runner bridge), but buildkit cannot reach this address because it runs in a separate docker container with its own network namespace.

Repro

ERROR: failed to build: failed to solve: Get "http://172.18.0.7:37871/_apis/artifactcache/cache?...": dial tcp 172.18.0.7:37871: i/o timeout

Hermes publish-image run 109: 17m39s failure. Openclaw publish-image run 66: 8m7s failure. Both at the cache-to/cache-from step.

Workaround in place

openclaw#8 and hermes#19 (merged) drop both cache lines. Loses cache perf — uncached builds take 6-8 minutes instead of 2-3.

Proper fix options

A. type=registry,ref=$IMAGE_NAME:buildcache — buildkit can pull/push to ECR over standard HTTPS. Works across container boundaries. Recommended.

B. type=inline — bakes cache info into image layers. Smaller cache but no extra registry round-trip. Good for templates with small dep churn.

C. Fix act_runner cache server to bind on the host network so buildkit containers can reach it. Upstream gitea/act_runner work.

Recommendation

Try option A (type=registry,ref=...:buildcache) on one template first (say openclaw) and measure rebuild time. If it works, roll out to other templates.

Acceptance criteria

  • publish-image rebuild time back to ~2-3 min for cached layer hits
  • Works in act_runner DinD topology
  • Cache eviction doesn't bloat ECR storage

Repos affected

All workspace templates with publish-image.yml:

  • molecule-ai-workspace-template-openclaw
  • molecule-ai-workspace-template-hermes
  • molecule-ai-workspace-template-deepagents
  • (presumably others)
## Problem publish-image.yml in workspace templates was using `cache-to: type=gha,mode=max` + `cache-from: type=gha` for buildkit cache. The act_runner provides a GHA-emulator cache server (e.g. at 172.18.0.7:37871 on the runner bridge), but buildkit cannot reach this address because it runs in a separate docker container with its own network namespace. ## Repro ERROR: failed to build: failed to solve: Get "http://172.18.0.7:37871/_apis/artifactcache/cache?...": dial tcp 172.18.0.7:37871: i/o timeout Hermes publish-image run 109: 17m39s failure. Openclaw publish-image run 66: 8m7s failure. Both at the cache-to/cache-from step. ## Workaround in place openclaw#8 and hermes#19 (merged) drop both cache lines. Loses cache perf — uncached builds take 6-8 minutes instead of 2-3. ## Proper fix options A. type=registry,ref=$IMAGE_NAME:buildcache — buildkit can pull/push to ECR over standard HTTPS. Works across container boundaries. Recommended. B. type=inline — bakes cache info into image layers. Smaller cache but no extra registry round-trip. Good for templates with small dep churn. C. Fix act_runner cache server to bind on the host network so buildkit containers can reach it. Upstream gitea/act_runner work. ## Recommendation Try option A (type=registry,ref=...:buildcache) on one template first (say openclaw) and measure rebuild time. If it works, roll out to other templates. ## Acceptance criteria - publish-image rebuild time back to ~2-3 min for cached layer hits - Works in act_runner DinD topology - Cache eviction doesn't bloat ECR storage ## Repos affected All workspace templates with publish-image.yml: - molecule-ai-workspace-template-openclaw - molecule-ai-workspace-template-hermes - molecule-ai-workspace-template-deepagents - (presumably others)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
approved
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 do-not-merge
hold from auto-merge (design review in progress)
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 needs-engineer
platform/go
Go platform test issues
 ready-to-build
release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1182
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?