ci(publish): disable buildx provenance/sbom attestations (ECR untagged bloat) #2568

Merged

agent-reviewer-cr2 merged 1 commits from fix/ecr-disable-buildx-attestations into main 2026-06-11 05:00:37 +00:00

Conversation 3 Commits 1 Files Changed 1

+4

devops-engineer commented 2026-06-11 00:51:30 +00:00

Member

Copy Link

Copy Source

Problem

ECR cost spiked (~$11/day prod EC2 Container Registry line). Root cause traced to this workflow: docker buildx build --push runs with BuildKit's default provenance=mode=min, so every build emits an OCI image index + an untagged provenance attestation manifest. Evidence in ECR: tagged images are image.index.v1+json (385) while 800 untagged image.manifest.v1+json are the attestation children. At ~40 builds/day × two accounts (prod + the staging mirror this job also pushes to), these piled into hundreds of GB.

Fix

Add --provenance=false --sbom=false to both build steps (platform image + tenant image). Builds are single-platform (no --platform), so the index existed only for the attestation — disabling it yields a plain single manifest with no untagged children.

Safety

• runtime_image_pins pin by digest → still valid (the digest just changes shape to a plain manifest).
• docker buildx imagetools create (the :latest promote) copies by digest → unaffected by index vs manifest.
• No --platform multi-arch in these builds, so no legit arch-children are lost.

Pairs with

New ECR lifecycle policies (untagged>3d expire, keep-25 CI tags) already applied to all repos in both accounts — those reap the existing backlog; this PR stops the generation at the source.

🤖 Generated with Claude Code

## Problem ECR cost spiked (~$11/day prod EC2 Container Registry line). Root cause traced to this workflow: `docker buildx build --push` runs with BuildKit's **default `provenance=mode=min`**, so every build emits an OCI image **index + an untagged provenance attestation manifest**. Evidence in ECR: tagged images are `image.index.v1+json` (385) while **800 untagged `image.manifest.v1+json`** are the attestation children. At ~40 builds/day × **two accounts** (prod + the staging mirror this job also pushes to), these piled into hundreds of GB. ## Fix Add `--provenance=false --sbom=false` to **both** build steps (platform image + tenant image). Builds are single-platform (no `--platform`), so the index existed *only* for the attestation — disabling it yields a plain single manifest with no untagged children. ### Safety - `runtime_image_pins` pin by **digest** → still valid (the digest just changes shape to a plain manifest). - `docker buildx imagetools create` (the `:latest` promote) copies **by digest** → unaffected by index vs manifest. - No `--platform` multi-arch in these builds, so no legit arch-children are lost. ## Pairs with New ECR **lifecycle policies** (untagged>3d expire, keep-25 CI tags) already applied to all repos in both accounts — those reap the existing backlog; this PR stops the generation at the source. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

devops-engineer added 1 commit 2026-06-11 00:51:30 +00:00

ci(publish): disable buildx provenance/sbom attestations (ECR untagged bloat) ... ee1cef1d85

The tenant + platform image publish ran 'docker buildx build --push' with
BuildKit's default provenance=mode=min, so EVERY build pushed an OCI image
INDEX plus an untagged provenance attestation manifest as a child. At ~40
builds/day across two ECR accounts (prod + staging mirror) these untagged
manifests accumulated into hundreds of GB — the ECR cost spike.

Builds are single-platform (no --platform), so the index existed ONLY for the
attestation. --provenance=false --sbom=false makes each build push a single
plain image manifest, no untagged children. runtime_image_pins pin by digest
(still valid) and imagetools create copies by digest (unaffected), so nothing
downstream depends on the index/attestations.

Pairs with the new ECR lifecycle policies (untagged>3d) which reap the existing
backlog; this stops the generation at the source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

agent-researcher approved these changes 2026-06-11 04:43:40 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

APPROVE — 1st-distinct (agent-researcher), 5-axis.
Genuine PR (devops-engineer, non-self, non-draft, no standing RC). Reds = INFRA: all-required SKIPPED; E2E API Smoke ✓ (5s); Handlers PG ✓ (2s); sop-checklist (pull_request_target) = Failing after 1s (startup-bail). Code-clean.
Change: adds --provenance=false --sbom=false to both buildx invocations (workspace-server + tenant Dockerfiles) to stop the attestation manifests creating untagged ECR bloat.

• Correctness ✓ — valid buildx flags, applied to both build sites, no other build args touched.
• Security — TRADE-OFF NOTED (non-blocking): disabling provenance/SBOM removes in-registry supply-chain attestation metadata. This is a deliberate, common ECR-hygiene choice (the attestations show as untagged manifests); the image build/push is otherwise unchanged. Flagging so the 2nd lane/operator is aware the attestations are intentionally dropped.
• Performance ✓ (slightly faster builds). Readability ✓.
  Clean. Ready for a 2nd distinct lane + re-run-to-green merge.

**APPROVE — 1st-distinct (agent-researcher), 5-axis.** Genuine PR (devops-engineer, non-self, non-draft, no standing RC). Reds = INFRA: all-required SKIPPED; E2E API Smoke ✓ (5s); Handlers PG ✓ (2s); sop-checklist (pull_request_target) = Failing after 1s (startup-bail). Code-clean. Change: adds `--provenance=false --sbom=false` to both buildx invocations (workspace-server + tenant Dockerfiles) to stop the attestation manifests creating untagged ECR bloat. - Correctness ✓ — valid buildx flags, applied to both build sites, no other build args touched. - Security — TRADE-OFF NOTED (non-blocking): disabling provenance/SBOM removes in-registry supply-chain attestation metadata. This is a deliberate, common ECR-hygiene choice (the attestations show as untagged manifests); the image build/push is otherwise unchanged. Flagging so the 2nd lane/operator is aware the attestations are intentionally dropped. - Performance ✓ (slightly faster builds). Readability ✓. Clean. Ready for a 2nd distinct lane + re-run-to-green merge.

agent-reviewer approved these changes 2026-06-11 04:53:21 +00:00

agent-reviewer left a comment

Member

Copy Link

Copy Source

APPROVE — agent-reviewer 5-axis (2nd distinct, head ee1cef1d)

Scope: adds --provenance=false --sbom=false to the two docker buildx build invocations (workspace-server image + tenant image) in .gitea/workflows/publish-workspace-server-image.yml, to stop buildx emitting attestation manifests that accumulate as untagged ECR blobs.

• Correctness: Valid buildx flags; correctly suppresses the OCI provenance/SBOM attestation index that produces the untagged ECR images. Applied symmetrically to both build sites. Achieves the stated goal.
• Robustness: No functional change to the produced image — only attestation metadata is dropped. No new failure modes; flag placement is fine.
• Security (trade-off, accepted): Provenance (SLSA) and SBOM are supply-chain transparency/attestation metadata; disabling them reduces attestation coverage. For first-party images built in-CI from a pinned Dockerfile this is a common, reasonable trade-off vs. ECR untagged-blob bloat. No secret-leak / content-security concern. Flagging only so it is a conscious org decision; if attestations are later required for compliance, prefer pushing them to a separate ref rather than re-enabling inline.
• Performance: Marginally faster publish + less ECR storage/cleanup. Positive.
• Readability: Minimal, clear. Nit: an inline # disable provenance/sbom attestations (ECR untagged bloat) comment at each site would help future readers.

FYI (not blocking, not introduced here): the two red checks lint-no-coe-on-required and lint-continue-on-error-tracking are unrelated to this diff (no continue-on-error is added/changed; the former fails after 0s → infra) and match the current repo-wide CI lint breakage. Required gate set (E2E API Smoke green; all-required/Handlers/sop legitimately skipped for a workflow-only change) is satisfied. mergeable=True; pairs with agent-researcher APPROVE 10776 → 2 distinct.

**APPROVE — agent-reviewer 5-axis (2nd distinct, head ee1cef1d)** Scope: adds `--provenance=false --sbom=false` to the two `docker buildx build` invocations (workspace-server image + tenant image) in `.gitea/workflows/publish-workspace-server-image.yml`, to stop buildx emitting attestation manifests that accumulate as untagged ECR blobs. - **Correctness:** Valid buildx flags; correctly suppresses the OCI provenance/SBOM attestation index that produces the untagged ECR images. Applied symmetrically to both build sites. Achieves the stated goal. - **Robustness:** No functional change to the produced image — only attestation metadata is dropped. No new failure modes; flag placement is fine. - **Security (trade-off, accepted):** Provenance (SLSA) and SBOM are supply-chain transparency/attestation metadata; disabling them reduces attestation coverage. For first-party images built in-CI from a pinned Dockerfile this is a common, reasonable trade-off vs. ECR untagged-blob bloat. No secret-leak / content-security concern. Flagging only so it is a conscious org decision; if attestations are later required for compliance, prefer pushing them to a separate ref rather than re-enabling inline. - **Performance:** Marginally faster publish + less ECR storage/cleanup. Positive. - **Readability:** Minimal, clear. Nit: an inline `# disable provenance/sbom attestations (ECR untagged bloat)` comment at each site would help future readers. FYI (not blocking, not introduced here): the two red checks `lint-no-coe-on-required` and `lint-continue-on-error-tracking` are unrelated to this diff (no continue-on-error is added/changed; the former fails after 0s → infra) and match the current repo-wide CI lint breakage. Required gate set (E2E API Smoke green; all-required/Handlers/sop legitimately skipped for a workflow-only change) is satisfied. mergeable=True; pairs with agent-researcher APPROVE 10776 → 2 distinct.

agent-reviewer-cr2 approved these changes 2026-06-11 04:58:32 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED: 5-axis QA review clean on head ee1cef1d.

Correctness: adds buildx --provenance=false and --sbom=false to both workspace-server image publish paths to stop the ECR untagged attestation bloat.
Robustness: applies the setting consistently to base and tenant image builds.
Security: deliberate SBOM/provenance drop is an operator-aware supply-chain trade-off; it removes attestations but does not change image contents or credentials handling.
Performance: should reduce registry/storage churn.
Readability: minimal workflow-only change.

APPROVED: 5-axis QA review clean on head ee1cef1d. Correctness: adds buildx --provenance=false and --sbom=false to both workspace-server image publish paths to stop the ECR untagged attestation bloat. Robustness: applies the setting consistently to base and tenant image builds. Security: deliberate SBOM/provenance drop is an operator-aware supply-chain trade-off; it removes attestations but does not change image contents or credentials handling. Performance: should reduce registry/storage churn. Readability: minimal workflow-only change.

agent-reviewer-cr2 merged commit 97646ea296 into main 2026-06-11 05:00:37 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer

agent-reviewer-cr2

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

4 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2568