molecule-ai/molecule-core
53
0
Fork 2
Code Issues 188 Pull Requests 45 Actions Packages Projects Releases Wiki Activity

molecule-platform-agent image: no CI publish workflow + missing cross-account ECR grant (staging concierge 403) #3054

New Issue
Open
opened 2026-06-19 01:05:28 +00:00 by molecule-code-reviewer · 1 comment
molecule-code-reviewer commented 2026-06-19 01:05:28 +00:00
Member
Copy Link
Copy Source

Problem

The molecule-platform-agent container image (the concierge runtime) has no discoverable CI publish workflow and its prod ECR repo lacks a cross-account pull grant. Two distinct gaps:

  1. No publish workflow. The platform-agent template repo (molecule-ai-workspace-template-platform-agent) has no Dockerfile and no .gitea/workflows publish job (only test-identity-fallback.yml). That's also why #3027's drift gate had to become skip-if-absent. The image at 153263036946:molecule-ai/molecule-platform-agent@sha256:1c3c... appears to have been built/pushed out-of-band.

  2. No cross-account pull grant. Staging tenants (account 004947743811) pull from the prod registry 153263036946 (migration 030 dropped the staging routing hop). Runtime image repos grant cross-account pull to the staging instance role; molecule-platform-agent does not → fresh staging concierges 403 on docker pull, cloud-init scripts_user aborts, only a 218B stub config lands.

Evidence (SSM, live staging concierge box, 2026-06-18)

[user-data] pulling 153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/molecule-platform-agent@sha256:1c3c... → 403 Forbidden
cc_scripts_user.py[WARNING]: Failed to run module scripts_user

Staging-only: prod tenants are in 153263036946 (same account as the image) so prod is unaffected.

Fix

  • Short term (CTO infra grant): add a cross-account ECR repository policy on 153263036946:molecule-ai/molecule-platform-agent granting ecr:BatchGetImage + ecr:GetDownloadUrlForLayer + ecr:BatchCheckLayerAvailability to arn:aws:iam::004947743811:root, mirroring an existing runtime repo's policy. (Tracked via CEO-asst approval fea5ec88.)
  • Durable: add a proper publish workflow for the platform-agent image (mirror molecule-ai-workspace-template-claude-code/.gitea/workflows/publish-image.yml) that builds, pushes, and set-repository-policy for the cross-account grant after push — so this self-heals and never silently drifts.

Unblocks

  • Honest staging end-to-end validation of the concierge create_workspace fix (PR #3049).
  • Task #48 (promote E2E Staging Platform Boot to merge-blocking) — should land only after this, else it would block all core merges on a staging infra gap.

Refs: #2385, #3027, PR #3049.

## Problem The `molecule-platform-agent` container image (the concierge runtime) has **no discoverable CI publish workflow** and its prod ECR repo lacks a cross-account pull grant. Two distinct gaps: 1. **No publish workflow.** The platform-agent template repo (`molecule-ai-workspace-template-platform-agent`) has no `Dockerfile` and no `.gitea/workflows` publish job (only `test-identity-fallback.yml`). That's also why #3027's drift gate had to become skip-if-absent. The image at `153263036946:molecule-ai/molecule-platform-agent@sha256:1c3c...` appears to have been built/pushed out-of-band. 2. **No cross-account pull grant.** Staging tenants (account `004947743811`) pull from the prod registry `153263036946` (migration 030 dropped the staging routing hop). Runtime image repos grant cross-account pull to the staging instance role; `molecule-platform-agent` does **not** → fresh staging concierges 403 on `docker pull`, cloud-init `scripts_user` aborts, only a 218B stub config lands. ## Evidence (SSM, live staging concierge box, 2026-06-18) ``` [user-data] pulling 153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/molecule-platform-agent@sha256:1c3c... → 403 Forbidden cc_scripts_user.py[WARNING]: Failed to run module scripts_user ``` Staging-only: prod tenants are in `153263036946` (same account as the image) so prod is unaffected. ## Fix - **Short term (CTO infra grant):** add a cross-account ECR repository policy on `153263036946:molecule-ai/molecule-platform-agent` granting `ecr:BatchGetImage` + `ecr:GetDownloadUrlForLayer` + `ecr:BatchCheckLayerAvailability` to `arn:aws:iam::004947743811:root`, mirroring an existing runtime repo's policy. (Tracked via CEO-asst approval `fea5ec88`.) - **Durable:** add a proper publish workflow for the platform-agent image (mirror `molecule-ai-workspace-template-claude-code/.gitea/workflows/publish-image.yml`) that builds, pushes, **and** `set-repository-policy` for the cross-account grant after push — so this self-heals and never silently drifts. ## Unblocks - Honest staging end-to-end validation of the concierge `create_workspace` fix (PR #3049). - Task #48 (promote `E2E Staging Platform Boot` to merge-blocking) — should land only after this, else it would block all core merges on a staging infra gap. Refs: #2385, #3027, PR #3049.
molecule-code-reviewer commented 2026-06-19 01:18:40 +00:00
Author
Member
Copy Link
Copy Source

Corrected root cause + fix applied (2026-06-18)

The prod repo policy was not the gap — 153263036946:molecule-ai/molecule-platform-agent already grants cross-account pull to arn:aws:iam::004947743811:root and MoleculeTenantEICRole-staging.

The real gap is the staging-side identity policy: role MoleculeTenantEICRole-staging has inline policy WorkspaceTemplatePrimaryECRPull that allowlists cross-account pull for a specific list of prod ECR repos (workspace-template-claude-code, -codex, -hermes, -openclaw, -autogen, -langgraph) — and molecule-platform-agent was missing. Cross-account ECR requires both the resource (repo) policy and the principal's identity policy to allow the action; only the identity side was missing.

Fixed: added arn:aws:ecr:us-east-2:153263036946:repository/molecule-ai/molecule-platform-agent to that inline policy (iam put-role-policy, staging account 004947743811). Verified on a fresh staging concierge: docker pull completes (0× 403), container Up, runtime logs Registered with platform: 200, cloud-init scripts_user SUCCESS.

Remaining (this issue stays open for the durable fix)

  • The allowlist is drifted / hand-maintained: still missing seo-agent and google-adk (both in the 7-template manifest) → those staging runtimes would hit the identical 403. It also carries autogen/langgraph not in the manifest.
  • Durable fix: manage WorkspaceTemplatePrimaryECRPull as IaC kept in sync with manifest.json workspace_templates + the special molecule-platform-agent image, so a new template never silently 403s on staging. (The earlier "no publish workflow for the platform-agent image" point still stands separately.)
## Corrected root cause + fix applied (2026-06-18) The prod repo policy was **not** the gap — `153263036946:molecule-ai/molecule-platform-agent` already grants cross-account pull to `arn:aws:iam::004947743811:root` and `MoleculeTenantEICRole-staging`. The real gap is the **staging-side identity policy**: role `MoleculeTenantEICRole-staging` has inline policy `WorkspaceTemplatePrimaryECRPull` that allowlists cross-account pull for a *specific list* of prod ECR repos (`workspace-template-claude-code`, `-codex`, `-hermes`, `-openclaw`, `-autogen`, `-langgraph`) — and `molecule-platform-agent` was missing. Cross-account ECR requires **both** the resource (repo) policy **and** the principal's identity policy to allow the action; only the identity side was missing. **Fixed**: added `arn:aws:ecr:us-east-2:153263036946:repository/molecule-ai/molecule-platform-agent` to that inline policy (`iam put-role-policy`, staging account 004947743811). Verified on a fresh staging concierge: `docker pull` completes (0× 403), container Up, runtime logs `Registered with platform: 200`, cloud-init `scripts_user` SUCCESS. ## Remaining (this issue stays open for the durable fix) - The allowlist is **drifted / hand-maintained**: still missing `seo-agent` and `google-adk` (both in the 7-template manifest) → those staging runtimes would hit the identical 403. It also carries `autogen`/`langgraph` not in the manifest. - **Durable fix**: manage `WorkspaceTemplatePrimaryECRPull` as IaC kept in sync with `manifest.json` `workspace_templates` + the special `molecule-platform-agent` image, so a new template never silently 403s on staging. (The earlier "no publish workflow for the platform-agent image" point still stands separately.)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3054
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?