test(e2e): template-asset delivery gate — fresh seo-agent must boot WITH skills (RFC #2843) #2971

Merged

core-devops merged 1 commits from feat/template-delivery-e2e-gate into main 2026-06-16 01:07:52 +00:00

Conversation 0 Commits 1 Files Changed 2

+246

core-devops commented 2026-06-15 22:17:36 +00:00

Member

Copy Link

Copy Source

RFC #2843 regression gate — fresh seo-agent must boot WITH its skills

Nothing in CI provisions a fresh agent end-to-end and inspects the delivered /configs. That blind spot is exactly why the 2026-06-15 incident family shipped green:

1. concierge booted with no model (MISSING_MODEL fail-closed) — fixed in #2966
2. concierge booted with no identity (generic Claude Code) — fix in #2955
3. seo-agent booted with config+prompts but no agent-skills/ — #32 (the seo-all skill pack, the whole point of the agent, is missing)

I confirmed #3 empirically: a fresh seo-agent provisioned on the current image gets config.yaml (8429 B real) + prompts/seo-agent.md + model, but /configs/agent-skills/ is empty.

What this adds

tests/e2e/test_template_delivery_e2e.sh — provisions a throwaway tenant + a fresh seo-agent template workspace, then asserts delivery, one assertion per incident:

	assertion	catches
A	seo-agent reaches online	MISSING_MODEL fail-closed
B	GET /model == declared model	model drop
C	config.yaml delivered + real (>1 KiB, not the 218 B stub)	config drop
D	prompts/ delivered	identity drop
E	agent-skills/seo-all/SKILL.md delivered	the #32 skill drop

Teardown trap deletes the org even on failure. Mirrors the auth/provision/teardown shape of test_staging_concierge_e2e.sh.

Staged rollout (intentional — do not flip to required yet)

• Phase 1 (this PR): advisory (continue-on-error), path-filtered to delivery code. It will report RED while #2955 and #32 are open — that's the point: it proves the gate correctly detects the bug.
• Phase 2 (after #2955 + #32 land and this is green ×2): drop continue-on-error and add to branch-protection required_status_checks → merge-blocking.

🤖 Generated with Claude Code

## RFC #2843 regression gate — fresh seo-agent must boot WITH its skills Nothing in CI provisions a fresh agent end-to-end and inspects the **delivered** `/configs`. That blind spot is exactly why the 2026-06-15 incident family shipped green: 1. concierge booted with **no model** (MISSING_MODEL fail-closed) — fixed in #2966 2. concierge booted with **no identity** (generic Claude Code) — fix in #2955 3. seo-agent booted with config+prompts but **no `agent-skills/`** — #32 (the seo-all skill pack, the whole point of the agent, is missing) I confirmed #3 empirically: a fresh `seo-agent` provisioned on the current image gets `config.yaml` (8429 B real) + `prompts/seo-agent.md` + model, but `/configs/agent-skills/` is **empty**. ### What this adds `tests/e2e/test_template_delivery_e2e.sh` — provisions a throwaway tenant + a fresh `seo-agent` template workspace, then asserts delivery, **one assertion per incident**: | | assertion | catches | |---|---|---| | A | seo-agent reaches `online` | MISSING_MODEL fail-closed | | B | `GET /model` == declared model | model drop | | C | `config.yaml` delivered + real (>1 KiB, not the 218 B stub) | config drop | | D | `prompts/` delivered | identity drop | | E | `agent-skills/seo-all/SKILL.md` delivered | **the #32 skill drop** | Teardown trap deletes the org even on failure. Mirrors the auth/provision/teardown shape of `test_staging_concierge_e2e.sh`. ### Staged rollout (intentional — do not flip to required yet) - **Phase 1 (this PR):** advisory (`continue-on-error`), path-filtered to delivery code. It will report **RED** while #2955 and #32 are open — that's the point: it proves the gate correctly *detects* the bug. - **Phase 2 (after #2955 + #32 land and this is green ×2):** drop `continue-on-error` and add to branch-protection `required_status_checks` → merge-blocking. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-devops added 1 commit 2026-06-15 22:17:38 +00:00

test(e2e): template-asset delivery gate — fresh seo-agent must boot WITH skills (RFC #2843 regression gate) ... 6c0a373572

Nothing in CI provisions a fresh agent end-to-end and inspects the DELIVERED
/configs — which is exactly why the 2026-06-15 incident family shipped green:
  1. concierge with no MODEL (MISSING_MODEL fail-closed)  — fixed #2966
  2. concierge with no IDENTITY (generic Claude Code)     — fix #2955
  3. seo-agent with config+prompts but NO agent-skills/   — #32

This adds tests/e2e/test_template_delivery_e2e.sh: provision a throwaway tenant
+ a fresh seo-agent template workspace, then assert the template-asset channel
actually delivered, with one assertion per incident:
  A. seo-agent reaches online                    (catches MISSING_MODEL)
  B. GET /model == declared model
  C. config.yaml delivered + REAL (>1KiB, not the 218B default stub)
  D. prompts/ delivered (identity prompt present)
  E. agent-skills/seo-all/SKILL.md delivered     (catches the #32 skill drop)
Teardown trap deletes the org even on failure.

Workflow template-delivery-e2e.yml is path-filtered (delivery code only) and
STAGED:
  Phase 1 (this PR): advisory (continue-on-error) — proves the gate correctly
          detects the open delivery bugs by going RED.
  Phase 2 (after #2955 + #32 land + green x2): drop continue-on-error and add
          to branch-protection required checks → merge-blocking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

core-devops referenced this issue from a commit 2026-06-16 00:33:10 +00:00

fix(image): build+push the molecule-platform-agent concierge image (fixes concierge identity never deploying)

core-devops referenced this pull request 2026-06-16 00:33:25 +00:00

fix(image): build+push the molecule-platform-agent concierge image (concierge identity) #2976

core-devops merged commit 2e31a8fc9d into main 2026-06-16 01:07:52 +00:00

core-devops referenced this issue from a commit 2026-06-16 01:31:16 +00:00

fix(image): auto-bump the platform-agent concierge image pin on every publish (true auto-bump)

core-devops referenced this issue from a commit 2026-06-16 02:17:51 +00:00

fix(image): build platform-agent FROM the live platform-tenant base (concierge identity image was failing on a dead base)

core-devops referenced this pull request 2026-06-16 02:18:03 +00:00

fix(image): build platform-agent FROM live platform-tenant base (unblocks concierge identity) #2982

agent-dev-b referenced this issue from a commit 2026-06-19 21:16:47 +00:00

ci(core#3081): A2A-probe concierge MCP tool list + promote creates-workspace to required

agent-dev-b referenced this issue from a commit 2026-06-19 22:09:40 +00:00

ci(core#3081): A2A-probe concierge MCP tool list + promote creates-workspace to required

agent-dev-b referenced this pull request 2026-06-19 22:10:25 +00:00

ci(core#3081): A2A-probe concierge MCP tool list + promote creates-workspace to required #3085

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2971