ci(ecr): auto-apply canonical image lifecycle policy on prod ECR pushes #3137

Merged

core-devops merged 1 commits from ops/ecr-lifecycle-iac into main 2026-06-22 01:23:50 +00:00

Conversation 2 Commits 1 Files Changed 3

+134

core-devops commented 2026-06-22 01:12:49 +00:00

Member

Copy Link

Copy Source

What

Make the prod ECR image lifecycle policy applied + maintained automatically by the publish pipelines (which already run with prod-ECR push creds), so the prod ECR storage bill (~$56/mo, account 153263036946) stops growing — without any standing prod-access grant.

Why

The prod ECR repos under 153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/* had lifecycle policies set out-of-band (no IaC managed them). Prod is bloated: platform-tenant alone has 70+ images / 12GB+ of untagged layers and superseded sha- tags that linger forever.

The publish workflows already aws ecr get-login-password + docker push to prod ECR — so they hold the right creds + region. Adding an aws ecr put-lifecycle-policy right after each push applies/refreshes the policy on every build. That's the durable IaC fix.

Changes

• scripts/ops/ensure-ecr-lifecycle.sh — shared, idempotent, fail-soft helper. The canonical lifecycle policy JSON is SSOT in this one file. put-lifecycle-policy only declares policy (no deletes — ECR's own lifecycle engine does the expiry on its schedule). Always exits 0 so a policy error (transient ECR blip, IAM gap) never breaks a publish — it logs a ::warning:: and the policy reapplies next publish.
• publish-workspace-server-image.yml — calls the script for molecule-ai/platform (after the platform push) and molecule-ai/platform-tenant (after the tenant push). Staging ECR (004947743811) is intentionally not touched.
• publish-canvas-image.yml — calls the script for molecule-ai/canvas after push.

Policy (canonical, validated on operator account)

• rule 1: expire untagged after 1 day
• rule 2: keep last 10 tagged for sha-/v/latest/staging/main prefixes

Verify

After merge, the next publish of each image applies the policy to its prod repo (the CI run is the verification). Out of band: aws ecr get-lifecycle-policy --repository-name molecule-ai/platform-tenant (needs prod creds).

Tested

• shellcheck clean; bash -n clean; embedded JSON parses to 2 rules.
• Fail-soft verified locally: no-arg, missing aws CLI, aws-returns-error, aws-success all exit 0 with correct logging.
• lint-curl-status-capture, lint-workflow-yaml, lint-publish-timeout all pass; existing scripts/ops unittest suite (34 tests) still green.

🤖 Generated with Claude Code

## What Make the prod ECR image lifecycle policy applied + maintained automatically by the publish pipelines (which already run with prod-ECR push creds), so the prod ECR storage bill (~$56/mo, account 153263036946) stops growing — without any standing prod-access grant. ## Why The prod ECR repos under `153263036946.dkr.ecr.us-east-2.amazonaws.com/molecule-ai/*` had lifecycle policies set **out-of-band** (no IaC managed them). Prod is bloated: `platform-tenant` alone has 70+ images / 12GB+ of untagged layers and superseded `sha-` tags that linger forever. The publish workflows **already** `aws ecr get-login-password` + `docker push` to prod ECR — so they hold the right creds + region. Adding an `aws ecr put-lifecycle-policy` right after each push applies/refreshes the policy on every build. That's the durable IaC fix. ## Changes - **`scripts/ops/ensure-ecr-lifecycle.sh`** — shared, idempotent, fail-soft helper. The canonical lifecycle policy JSON is **SSOT in this one file**. `put-lifecycle-policy` only **declares** policy (no deletes — ECR's own lifecycle engine does the expiry on its schedule). Always exits 0 so a policy error (transient ECR blip, IAM gap) **never breaks a publish** — it logs a `::warning::` and the policy reapplies next publish. - **`publish-workspace-server-image.yml`** — calls the script for `molecule-ai/platform` (after the platform push) and `molecule-ai/platform-tenant` (after the tenant push). Staging ECR (`004947743811`) is intentionally not touched. - **`publish-canvas-image.yml`** — calls the script for `molecule-ai/canvas` after push. ## Policy (canonical, validated on operator account) - rule 1: expire untagged after 1 day - rule 2: keep last 10 tagged for `sha-`/`v`/`latest`/`staging`/`main` prefixes ## Verify After merge, the next publish of each image applies the policy to its prod repo (the CI run is the verification). Out of band: `aws ecr get-lifecycle-policy --repository-name molecule-ai/platform-tenant` (needs prod creds). ## Tested - shellcheck clean; `bash -n` clean; embedded JSON parses to 2 rules. - Fail-soft verified locally: no-arg, missing aws CLI, aws-returns-error, aws-success all exit 0 with correct logging. - `lint-curl-status-capture`, `lint-workflow-yaml`, `lint-publish-timeout` all pass; existing `scripts/ops` unittest suite (34 tests) still green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-22 01:12:51 +00:00

ci(ecr): auto-apply canonical image lifecycle policy on prod ECR pushes ... 5a54d34c75

The prod ECR repos under 153263036946.dkr.ecr.us-east-2.amazonaws.com/
molecule-ai/* had lifecycle policies set out-of-band (no IaC), so the
prod ECR storage bill (~$56/mo) kept growing — platform-tenant alone
accumulated 70+ images / 12GB+ of untagged + superseded sha- tags.

Durable fix: the publish workflows already authenticate to prod ECR and
push images (right creds + region), so apply the lifecycle policy right
after each push. ECR's lifecycle engine then expires old images on its
own schedule — this only DECLARES policy, no deletes happen here.

- scripts/ops/ensure-ecr-lifecycle.sh: shared, idempotent, fail-soft
  helper (always exit 0 so a policy error never breaks a publish). The
  canonical policy JSON is SSOT in this one file: expire untagged after
  1 day; keep last 10 tagged for sha-/v/latest/staging/main prefixes.
- publish-workspace-server-image.yml: apply to molecule-ai/platform
  (after platform push) + molecule-ai/platform-tenant (after tenant push).
- publish-canvas-image.yml: apply to molecule-ai/canvas after push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

molecule-code-reviewer approved these changes 2026-06-22 01:23:46 +00:00

molecule-code-reviewer left a comment

Member

Copy Link

Copy Source

Reviewed: additive post-push ensure-ecr-lifecycle step, fail-soft (never breaks publish), canonical policy SSOT, lints pass. Durable prod-ECR cost guard. LGTM.

Reviewed: additive post-push ensure-ecr-lifecycle step, fail-soft (never breaks publish), canonical policy SSOT, lints pass. Durable prod-ECR cost guard. LGTM.

core-security approved these changes 2026-06-22 01:23:48 +00:00

core-security left a comment

Member

Copy Link

Copy Source

Reviewed: additive post-push ensure-ecr-lifecycle step, fail-soft (never breaks publish), canonical policy SSOT, lints pass. Durable prod-ECR cost guard. LGTM.

Reviewed: additive post-push ensure-ecr-lifecycle step, fail-soft (never breaks publish), canonical policy SSOT, lints pass. Durable prod-ECR cost guard. LGTM.

core-devops scheduled this pull request to auto merge when all checks succeed 2026-06-22 01:23:49 +00:00

core-devops merged commit b73723b3f4 into main 2026-06-22 01:23:50 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

molecule-code-reviewer

core-security

Labels

Clear labels

approved

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

do-not-merge

hold from auto-merge (design review in progress)

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

needs-engineer

platform/go

Go platform test issues

ready-to-build

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3137