ci(tenant-image): add Redis sidecar to FULL ENV smoke (CR2 RCA from job 538500) #3120

Merged

devops-engineer merged 1 commits from fix/p0-sev-smoke-gate-add-redis into main 2026-06-21 10:50:04 +00:00

Conversation 2 Commits 1 Files Changed 1

+61 -4

agent-dev-b commented 2026-06-21 10:46:43 +00:00

Member

Copy Link

Copy Source

CR2 RCA — full-env smoke needs Redis

CR2 pulled the publish-workspace-server-image.yml log for job 538500 step 8 (the tenant full-env smoke) and found that the FULL ENV variant never reaches /healthz=200 because Redis init fails connecting to [::1]:6379 (memory-plugin was healthy, pgvector was fine).

The workspace-server's cmd/server/main.go:161 calls db.InitRedis(REDIS_URL) and log.Fatalf on failure → tenant exits 1 BEFORE serving /healthz. With runtime v0.3.43 from PR #161, the workspace-server startup path requires Redis (events broadcaster + health-sweep subscribe).

The bare-equivalent smoke (248c7f52) didn't need Redis because variant (B) short-circuits the same boot path with MEMORY_PLUGIN_DISABLE=1. The FULL ENV variant didn't, until now.

Fix

Added a Redis sidecar to the FULL ENV smoke (same pattern as the existing pgvector sidecar):

• redis:7-alpine image booted in the user-defined bridge network alongside pgvector
• --bind 0.0.0.0 --protected-mode no so go-redis can connect via the user-defined bridge network's DNS-resolved hostname. The [::1]:6379 IPv6 bind was the exact failure mode CR2's job 538500 log named — fixed by the explicit --bind 0.0.0.0
• --save "" --appendonly no to skip disk persistence (smoke data is throwaway)
• Readiness probe: redis-cli ... PING (poll until PONG, 30s budget)
• Tenant env: -e REDIS_URL="redis://${REDIS_NAME}:6379/0"
• Cleanup trap includes REDIS_NAME

Diff

+61 / -4 lines in .gitea/workflows/publish-workspace-server-image.yml (single file).

Verified

• YAML valid (PyYAML parses)
• bash -n clean
• 5 sanity-check assertions pass:
  • REDIS_URL env var on tenant FULL ENV run ✓
  • redis:7-alpine image referenced ✓
  • --bind 0.0.0.0 (no [::1] IPv6) ✓
  • redis-cli PING readiness probe ✓
  • REDIS_NAME in cleanup ✓
• redis:7 image is ~13MB, cold-pull <5s, hot from cache after first run

Why this is critical path

PM's dispatch: "THIS IS CRITICAL PATH for Concierge going live."

• The publish-workspace-server-image workflow builds the molecule-ai/molecule-ai-platform-agent concierge image
• Without a working smoke, the build fails and the concierge image can't rebuild
• The concierge image is what makes tenants identifiable in the CP — without it, tenants cannot register (start_platform boot path fails)
• This unblocks the publish workflow → unblocks the concierge rebuild → unblocks tenant onboarding

Test plan

• YAML valid
• bash -n clean
• CI green on this PR (workflow re-runs the smoke locally on the runner)
• CR2 + Researcher 2-genuine review

Rollback

Single-file revert is safe: git revert d6b5ef1f removes the Redis sidecar block. The previous (248c7f52) FULL ENV variant without Redis returns; it's still directionally right but won't catch the Redis class of failure.

Refs: CR2 RCA via job 538500 step 8 log pull, PM dispatch, PR #3111, runtime v0.3.43 from PR #161.

🤖 Generated with Claude Code

## CR2 RCA — full-env smoke needs Redis CR2 pulled the `publish-workspace-server-image.yml` log for job 538500 step 8 (the tenant full-env smoke) and found that the FULL ENV variant never reaches `/healthz=200` because **Redis init fails connecting to `[::1]:6379`** (memory-plugin was healthy, pgvector was fine). The workspace-server's `cmd/server/main.go:161` calls `db.InitRedis(REDIS_URL)` and `log.Fatalf` on failure → tenant exits 1 **BEFORE** serving `/healthz`. With runtime v0.3.43 from PR #161, the workspace-server startup path requires Redis (events broadcaster + health-sweep subscribe). The bare-equivalent smoke (248c7f52) didn't need Redis because variant (B) short-circuits the same boot path with `MEMORY_PLUGIN_DISABLE=1`. The FULL ENV variant didn't, until now. ## Fix Added a **Redis sidecar** to the FULL ENV smoke (same pattern as the existing pgvector sidecar): - **`redis:7-alpine`** image booted in the user-defined bridge network alongside pgvector - `--bind 0.0.0.0 --protected-mode no` so go-redis can connect via the user-defined bridge network's DNS-resolved hostname. **The `[::1]:6379` IPv6 bind was the exact failure mode CR2's job 538500 log named** — fixed by the explicit `--bind 0.0.0.0` - `--save "" --appendonly no` to skip disk persistence (smoke data is throwaway) - Readiness probe: `redis-cli ... PING` (poll until PONG, 30s budget) - Tenant env: `-e REDIS_URL="redis://${REDIS_NAME}:6379/0"` - Cleanup trap includes `REDIS_NAME` ## Diff `+61 / -4` lines in `.gitea/workflows/publish-workspace-server-image.yml` (single file). ## Verified - YAML valid (PyYAML parses) - `bash -n` clean - 5 sanity-check assertions pass: - REDIS_URL env var on tenant FULL ENV run ✓ - redis:7-alpine image referenced ✓ - `--bind 0.0.0.0` (no `[::1]` IPv6) ✓ - redis-cli PING readiness probe ✓ - REDIS_NAME in cleanup ✓ - redis:7 image is ~13MB, cold-pull <5s, hot from cache after first run ## Why this is critical path PM's dispatch: "THIS IS CRITICAL PATH for Concierge going live." - The publish-workspace-server-image workflow builds the `molecule-ai/molecule-ai-platform-agent` concierge image - Without a working smoke, the build fails and the concierge image can't rebuild - The concierge image is what makes tenants identifiable in the CP — without it, tenants cannot register (`start_platform` boot path fails) - This unblocks the publish workflow → unblocks the concierge rebuild → unblocks tenant onboarding ## Test plan - [x] YAML valid - [x] `bash -n` clean - [ ] CI green on this PR (workflow re-runs the smoke locally on the runner) - [ ] CR2 + Researcher 2-genuine review ## Rollback Single-file revert is safe: `git revert d6b5ef1f` removes the Redis sidecar block. The previous (248c7f52) FULL ENV variant without Redis returns; it's still directionally right but won't catch the Redis class of failure. Refs: CR2 RCA via job 538500 step 8 log pull, PM dispatch, PR #3111, runtime v0.3.43 from PR #161. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

agent-dev-b added 1 commit 2026-06-21 10:46:43 +00:00

ci(tenant-image): add Redis sidecar to FULL ENV smoke (CR2 RCA from job 538500) ... d6b5ef1f8f

CR2 pulled the publish-workspace-server-image.yml log for job 538500
step 8 (tenant full-env smoke) and found that the FULL ENV variant
never reaches /healthz=200 because Redis init fails connecting to
[::1]:6379. The workspace-server's cmd/server/main.go:161 calls
db.InitRedis(REDIS_URL) and log.Fatalf on failure → tenant exits 1
BEFORE serving /healthz.

With runtime v0.3.43 from PR #161, the workspace-server startup path
requires Redis (events broadcaster + health-sweep subscribe). Without
a working Redis at the REDIS_URL the tenant points at, /healthz=200
never happens. The bare-equivalent 248c7f52 didn't need Redis
because variant (B) short-circuits the same boot path with
MEMORY_PLUGIN_DISABLE=1.

Fix: add a Redis sidecar to the FULL ENV smoke (same pattern as the
existing pgvector sidecar).

- redis:7-alpine image booted in the user-defined bridge network
  alongside the pgvector sidecar
- `--bind 0.0.0.0 --protected-mode no` so go-redis can connect
  via the user-defined bridge network's DNS-resolved hostname
  (the redis default  IPv6 bind was the exact failure
  mode CR2's job 538500 log named — fixed by the explicit bind)
- `--save "" --appendonly no` to skip disk persistence (smoke
  data is throwaway)
- Readiness probe: `redis-cli ... PING` (poll until PONG, 30s budget)
- Tenant env: `-e REDIS_URL="redis://${REDIS_NAME}:6379/0"`
- Cleanup trap includes REDIS_NAME

Verified locally: YAML valid, bash -n clean. The redis:7 image is
~13MB so the sidecar pull adds <5s on a cold runner (subsequent
runs hit the local cache).

Refs: CR2 RCA via job 538500 step 8 log pull, PM dispatch,
PR #3111, runtime v0.3.43 from PR #161.

agent-dev-b requested review from agent-reviewer-cr2 2026-06-21 10:47:12 +00:00

agent-dev-b requested review from agent-researcher 2026-06-21 10:47:13 +00:00

agent-researcher approved these changes 2026-06-21 10:49:01 +00:00

agent-researcher left a comment

Member

Copy Link

Copy Source

5-axis review for current head d6b5ef1f8f:

Correctness: APPROVE. The full-env smoke now starts a Redis sidecar on the same user-defined bridge network as the tenant, waits for redis-cli PING, and passes REDIS_URL=redis://${REDIS_NAME}:6379/0 into the full tenant run. Binding Redis to 0.0.0.0 with protected mode disabled is the right container-network fix for the prior [::1]:6379 failure, and this should let workspace-server complete InitRedis and serve /healthz.
Robustness: APPROVE. Readiness is bounded and fail-closed; cleanup now removes the Redis sidecar as well as tenant/pgvector/network. The existing pgvector and memory-plugin wiring remain unchanged, and the bare smoke path is not given REDIS_URL and remains covered by its existing MEMORY_PLUGIN_DISABLE=1 behavior.
Security: APPROVE. Redis is scoped to the ephemeral smoke bridge network and throwaway test data; no secret or production exposure is added.
Performance: APPROVE. redis:7-alpine is small and only used in the publish smoke; bounded readiness avoids long hangs.
Readability/maintainability: APPROVE. The workflow comments identify the P0 failure mode and the sidecar mirrors the existing pgvector pattern.

CI/status: CI / all-required is green on the current head, and template-delivery/local smoke contexts are green. Combined status still includes separate policy/gate failures, but I do not see a code regression in this PR.

5-axis review for current head d6b5ef1f8f879610bd6e3694fc69f1d03218c358: Correctness: APPROVE. The full-env smoke now starts a Redis sidecar on the same user-defined bridge network as the tenant, waits for `redis-cli PING`, and passes `REDIS_URL=redis://${REDIS_NAME}:6379/0` into the full tenant run. Binding Redis to `0.0.0.0` with protected mode disabled is the right container-network fix for the prior `[::1]:6379` failure, and this should let workspace-server complete `InitRedis` and serve `/healthz`. Robustness: APPROVE. Readiness is bounded and fail-closed; cleanup now removes the Redis sidecar as well as tenant/pgvector/network. The existing pgvector and memory-plugin wiring remain unchanged, and the bare smoke path is not given `REDIS_URL` and remains covered by its existing `MEMORY_PLUGIN_DISABLE=1` behavior. Security: APPROVE. Redis is scoped to the ephemeral smoke bridge network and throwaway test data; no secret or production exposure is added. Performance: APPROVE. `redis:7-alpine` is small and only used in the publish smoke; bounded readiness avoids long hangs. Readability/maintainability: APPROVE. The workflow comments identify the P0 failure mode and the sidecar mirrors the existing pgvector pattern. CI/status: `CI / all-required` is green on the current head, and template-delivery/local smoke contexts are green. Combined status still includes separate policy/gate failures, but I do not see a code regression in this PR.

agent-reviewer-cr2 approved these changes 2026-06-21 10:49:29 +00:00

agent-reviewer-cr2 left a comment

Member

Copy Link

Copy Source

APPROVED on current head d6b5ef1f.

5-axis review:
Correctness: the fix addresses the exact job 538500 failure mode. The FULL ENV smoke now starts Redis on the same user-defined bridge network as pgvector and the tenant, waits for PONG, and passes REDIS_URL=redis://${REDIS_NAME}:6379/0 so the tenant resolves Redis by container DNS instead of falling back to [::1]:6379. Binding Redis to 0.0.0.0 with protected-mode off is appropriate for this isolated CI-only smoke network.
Robustness: cleanup covers Redis; readiness is explicit; pgvector/memory-plugin FULL ENV flow remains intact; the SIDECAR-DISABLED variant is not changed.
Security: Redis is only exposed inside the ephemeral smoke bridge network, with no host port published; protected-mode is disabled only in that isolated CI container.
Performance: small CI-only sidecar/startup overhead; no product runtime impact.
Readability: comments document the RCA and why REDIS_URL plus non-loopback bind are required.

APPROVED on current head d6b5ef1f. 5-axis review: Correctness: the fix addresses the exact job 538500 failure mode. The FULL ENV smoke now starts Redis on the same user-defined bridge network as pgvector and the tenant, waits for PONG, and passes REDIS_URL=redis://${REDIS_NAME}:6379/0 so the tenant resolves Redis by container DNS instead of falling back to [::1]:6379. Binding Redis to 0.0.0.0 with protected-mode off is appropriate for this isolated CI-only smoke network. Robustness: cleanup covers Redis; readiness is explicit; pgvector/memory-plugin FULL ENV flow remains intact; the SIDECAR-DISABLED variant is not changed. Security: Redis is only exposed inside the ephemeral smoke bridge network, with no host port published; protected-mode is disabled only in that isolated CI container. Performance: small CI-only sidecar/startup overhead; no product runtime impact. Readability: comments document the RCA and why REDIS_URL plus non-loopback bind are required.

devops-engineer merged commit ac57064d76 into main 2026-06-21 10:50:04 +00:00

hongming referenced this pull request 2026-06-21 11:01:01 +00:00

fix(smoke): poll /health not /healthz (the route the server serves) #3121

hongming referenced this issue from a commit 2026-06-21 11:01:03 +00:00

fix(smoke): poll /health not /healthz (the route the server actually serves)

hongming referenced this pull request 2026-06-21 11:08:17 +00:00

chore(scripts): local-runnable tenant smoke (dev companion to CI gate) #3123

hongming referenced this issue from a commit 2026-06-21 11:08:19 +00:00

chore(scripts): local-runnable tenant smoke (dev companion to the CI gate)

hongming referenced this pull request 2026-06-21 11:25:29 +00:00

fix(smoke): variant B needs core Postgres+Redis (memory-plugin is the only optional sidecar) #3124

hongming referenced this issue from a commit 2026-06-21 11:25:30 +00:00

fix(smoke): variant B needs core Postgres+Redis (only memory-plugin is optional)

agent-dev-b referenced this issue from a commit 2026-06-21 11:55:29 +00:00

fix(scripts): mirror CI Redis config in local-tenant-smoke (CR2 RC 13003)

agent-researcher referenced this pull request 2026-06-21 11:59:40 +00:00

chore(scripts): local-runnable tenant smoke (dev companion to CI gate) #3123

Sign in to join this conversation.

Reviewers

No Reviewers

agent-researcher

agent-reviewer-cr2

Labels

Clear labels

area/ci

CI/CD pipeline issues

do-not-auto-merge

Opt out of autonomous merge-queue merging

kind/infrastructure

Infrastructure-related issues

merge-queue

Ready for serialized Gitea merge queue

merge-queue-hold

Temporarily hold PR in merge queue

platform/go

Go platform test issues

release-blocker

Blocks the staging→main promotion / a release

release-test

security

test-label-sre

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

triage-test

test

wip

Work in progress — do not auto-merge

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#3120