fix(plugins): SaaS (EC2-per-workspace) install/uninstall via EIC SSH #84

Merged

claude-ceo-assistant merged 5 commits from fix/saas-plugin-install-eic into staging 2026-05-08 02:15:50 +00:00

Conversation 1 Commits 5 Files Changed 8 +967 -30

claude-ceo-assistant commented 2026-05-07 22:43:46 +00:00

Owner

Copy Link

Summary

Closes the 🔴 docker-only row in docs/architecture/backends.md:57. Plugin install on every SaaS tenant currently 503s with workspace container not running. Caught on hongming.moleculesai.app Claude Code Agent (workspace c7244ed9-f623-4cba-8873-020e5c9fe104) when canvas POST /workspaces//plugins surfaced the error.

Root cause

PluginsHandler.deliverToContainer (and Uninstall) require a local Docker container to exec into. On SaaS tenants MOLECULE_ORG_ID is set → prov is nil → dockerCli is nil → findRunningContainer returns "" → 503. The workspace container actually lives on its own EC2 (workspaces.instance_id is populated), so even mounting the docker socket on the tenant box wouldn't help — the container isn't on this host.

Approach

Mirror the Files API PR #1702 pattern. New plugins_install_eic.go adds three EIC-tunnel-backed primitives (installPluginViaEIC, uninstallPluginViaEIC, readPluginManifestViaEIC) that reuse the existing withEICTunnel primitive from template_files_eic.go. deliverToContainer and Uninstall now dispatch:

1. h.docker != nil AND local container up → existing exec+cp path.
2. instance_id set → push the staged plugin tarball over SSH stdin into the EC2's bind-mounted /configs/plugins/<name>/ (per workspaceFilePathPrefix), chown 1000:1000, restart.
3. runtime == "external" → 422 with hint pointing at the Download endpoint (pre-existing guard, unchanged).
4. Else 503.

Direct host write (rather than docker-cp via SSH) because the runtime's config dir is already bind-mounted into the workspace container — the runtime sees the files on next start with no additional plumbing.

Why a new lookup

InstanceIDLookup (parallel to the existing RuntimeLookup) lets unit tests drive the SaaS path without a DB. Production wires it in router.go against the workspaces.instance_id column the same way templates.go and terminal.go do.

Tests

• TestPluginInstall_SaaS_DispatchesToEIC — full Install pipeline with stubbed EIC sees the staged tarball + correct (instance, runtime, plugin) tuple.
• TestPluginInstall_SaaS_PropagatesEICError — EIC failure surfaces 502, response body doesn't echo raw ssh stderr.
• TestPluginInstall_NoBackends_Returns503 — empty instance_id + nil docker → 503 (not silent dispatch).
• TestPluginInstall_InstanceLookupError_Returns503 — DB hiccup on lookup fails open to 503.
• TestPluginUninstall_SaaS_DispatchesToEIC + _PropagatesEICError + _NoBackends_Returns503 — symmetric uninstall coverage.
• TestBuildPlugin{Install,Uninstall,ManifestRead}Shell_QuotesPath — pure shell-shape regression pins.
• TestHostPluginPath_PerRuntime — claude-code/hermes/langgraph/unknown-runtime fallback paths.
• TestRealInstallPluginViaEIC_TarPayloadShape — tar.gz round-trip catches streamDirAsTar regression.

All existing handler / dispatcher / architecture-gate tests stay green:

ok  internal/handlers  7.035s

Test plan after merge

• Watch staging deploy and gh pr checks (Gitea Actions).
• On staging tenant: provision a claude-code workspace, install browser-automation via canvas, observe /configs/plugins/browser-automation/ on the workspace EC2, confirm plugin shows up after the auto-restart.
• Production: after staging-pass redeploy chain completes, install browser-automation on the Claude Code Agent workspace on hongming.moleculesai.app. Confirm 200 response + plugin visible after restart.

Out of scope

• Mirroring docker-path's eager /configs/skills/<skill> cleanup on uninstall over SSH. The runtime adapter rewrites /configs/skills/ from the live plugin set on restart, so a stale skill dir self-cleans. Two extra ssh round-trips per uninstall would be churn for no behavioural win; can revisit if a real bug surfaces.
• Mirroring docker-path's CLAUDE.md awk-strip on uninstall. Same reason — runtime adapter rewrites that file on restart.
• A new CP-level SSM-exec primitive. Not needed since EIC is already plumbed and the runtime config dir is host-mounted.

Rollback

One PR. No DB migration, no new env vars. Existing Docker-mode path is untouched.

## Summary Closes the 🔴 docker-only row in `docs/architecture/backends.md:57`. Plugin install on every SaaS tenant currently 503s with `workspace container not running`. Caught on `hongming.moleculesai.app` Claude Code Agent (workspace `c7244ed9-f623-4cba-8873-020e5c9fe104`) when canvas POST /workspaces/<id>/plugins surfaced the error. ## Root cause `PluginsHandler.deliverToContainer` (and `Uninstall`) require a local Docker container to exec into. On SaaS tenants `MOLECULE_ORG_ID` is set → `prov` is nil → `dockerCli` is nil → `findRunningContainer` returns `""` → 503. The workspace container actually lives on its own EC2 (`workspaces.instance_id` is populated), so even mounting the docker socket on the tenant box wouldn't help — the container isn't on this host. ## Approach Mirror the Files API PR #1702 pattern. New `plugins_install_eic.go` adds three EIC-tunnel-backed primitives (`installPluginViaEIC`, `uninstallPluginViaEIC`, `readPluginManifestViaEIC`) that reuse the existing `withEICTunnel` primitive from `template_files_eic.go`. `deliverToContainer` and `Uninstall` now dispatch: 1. `h.docker != nil` AND local container up → existing exec+cp path. 2. `instance_id` set → push the staged plugin tarball over SSH stdin into the EC2's bind-mounted `/configs/plugins/<name>/` (per `workspaceFilePathPrefix`), `chown 1000:1000`, restart. 3. `runtime == "external"` → 422 with hint pointing at the Download endpoint (pre-existing guard, unchanged). 4. Else 503. Direct host write (rather than docker-cp via SSH) because the runtime's config dir is already bind-mounted into the workspace container — the runtime sees the files on next start with no additional plumbing. ## Why a new lookup `InstanceIDLookup` (parallel to the existing `RuntimeLookup`) lets unit tests drive the SaaS path without a DB. Production wires it in `router.go` against the `workspaces.instance_id` column the same way `templates.go` and `terminal.go` do. ## Tests - `TestPluginInstall_SaaS_DispatchesToEIC` — full Install pipeline with stubbed EIC sees the staged tarball + correct (instance, runtime, plugin) tuple. - `TestPluginInstall_SaaS_PropagatesEICError` — EIC failure surfaces 502, response body doesn't echo raw ssh stderr. - `TestPluginInstall_NoBackends_Returns503` — empty `instance_id` + nil docker → 503 (not silent dispatch). - `TestPluginInstall_InstanceLookupError_Returns503` — DB hiccup on lookup fails open to 503. - `TestPluginUninstall_SaaS_DispatchesToEIC` + `_PropagatesEICError` + `_NoBackends_Returns503` — symmetric uninstall coverage. - `TestBuildPlugin{Install,Uninstall,ManifestRead}Shell_QuotesPath` — pure shell-shape regression pins. - `TestHostPluginPath_PerRuntime` — claude-code/hermes/langgraph/unknown-runtime fallback paths. - `TestRealInstallPluginViaEIC_TarPayloadShape` — tar.gz round-trip catches `streamDirAsTar` regression. All existing handler / dispatcher / architecture-gate tests stay green: ``` ok internal/handlers 7.035s ``` ## Test plan after merge - [ ] Watch staging deploy and `gh pr checks` (Gitea Actions). - [ ] On staging tenant: provision a claude-code workspace, install `browser-automation` via canvas, observe `/configs/plugins/browser-automation/` on the workspace EC2, confirm plugin shows up after the auto-restart. - [ ] Production: after staging-pass redeploy chain completes, install `browser-automation` on the Claude Code Agent workspace on `hongming.moleculesai.app`. Confirm 200 response + plugin visible after restart. ## Out of scope - Mirroring docker-path's eager `/configs/skills/<skill>` cleanup on uninstall over SSH. The runtime adapter rewrites `/configs/skills/` from the live plugin set on restart, so a stale skill dir self-cleans. Two extra ssh round-trips per uninstall would be churn for no behavioural win; can revisit if a real bug surfaces. - Mirroring docker-path's `CLAUDE.md` awk-strip on uninstall. Same reason — runtime adapter rewrites that file on restart. - A new CP-level SSM-exec primitive. Not needed since EIC is already plumbed and the runtime config dir is host-mounted. ## Rollback One PR. No DB migration, no new env vars. Existing Docker-mode path is untouched.

claude-ceo-assistant added 1 commit 2026-05-07 22:43:48 +00:00

fix(plugins): SaaS (EC2-per-workspace) install/uninstall via EIC SSH ... 16868c4ec1

Closes the 🔴 docker-only row in docs/architecture/backends.md. Plugin
install on every SaaS tenant currently 503s with "workspace container
not running" because the handler is hardcoded to Docker exec but SaaS
workspaces live on per-workspace EC2s. Caught on hongming.moleculesai.app
when canvas POST /workspaces/<id>/plugins surfaced the error.

Mirrors the Files API PR #1702 pattern: dispatch on workspaces.instance_id
in deliverToContainer (and Uninstall). When set, push the staged plugin
tarball to the EC2 over the existing withEICTunnel primitive
(template_files_eic.go) and unpack into the runtime's bind-mounted config
dir (/configs for claude-code, /home/ubuntu/.hermes for hermes — see
workspaceFilePathPrefix). chown 1000:1000 to match the docker path's
agent-uid contract; restart via the existing dispatcher.

Direct host write rather than docker-cp via SSH because the runtime's
config dir is already bind-mounted into the workspace container — the
runtime sees the files on next start with no additional plumbing.

Adds InstanceIDLookup (parallel to RuntimeLookup) so unit tests don't
need a DB; production wires it in router.go like templates.go does.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

claude-ceo-assistant referenced this issue from a commit 2026-05-07 23:01:15 +00:00

fix(eic-tunnel-pool): capture poolJanitorInterval at pool construction

claude-ceo-assistant added 1 commit 2026-05-07 23:01:15 +00:00

fix(eic-tunnel-pool): capture poolJanitorInterval at pool construction ... b664691051

Closes the chronic -race flake on TestPooledWithEICTunnel_PanicPoisonsEntry
and the handlers package as a whole (CI / Platform (Go) was intermittent
on staging, ~50% red on workspace-server-touching commits since 2026-04).

The race: tests swap the package-level poolJanitorInterval via t.Cleanup
(eic_tunnel_pool_test.go:61) AFTER an earlier test caused the global pool's
janitor goroutine to start. The janitor loops on time.NewTicker(poolJanitorInterval)
on every tick — so the cleanup write races the goroutine read for the rest
of the process. Caught locally + on PR #84's CI run on Gitea.

Fix: capture the interval as a field on eicTunnelPool at newEICTunnelPool().
The janitor now reads p.janitorInterval, which never changes after construction.
Tests that override poolJanitorInterval before freshPool() still get the new
value (they set the package var before construction). The global pool's
janitor — created lazily once via sync.Once on first getEICTunnelPool() —
is now immune to t.Cleanup-driven swaps from later tests.

Surfaced while verifying #84 (SaaS plugin install via EIC SSH); folded
into this PR per the "fix root not symptom" rule rather than merging
around a chronic-red CI signal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

hongming referenced this pull request 2026-05-07 23:35:05 +00:00

CI: 3 chronic Gitea-Actions workflow flakes (pr-guards / Harness Replays / handlers-postgres-integration) #88

hongming commented 2026-05-07 23:35:29 +00:00

Owner

Copy Link

CI verdict

Functional checks all green: CI / Platform (Go), Handlers-Postgres-Integration (on the 1st commit), E2E API Smoke, Playwright, CodeQL, Canvas, Python, Runtime PR-Built, Secret-scan. Total 23/26 success.

The 3 reds are pre-existing Gitea-vs-GitHub Actions infrastructure issues — none from this PR's code path. Verified by reproducing each failure on origin/staging directly:

• ❌ pr-guards / disable-auto-merge-on-push — gh CLI calls GitHub's GraphQL → Gitea returns 405. Fails on every Gitea PR push since the migration. Workflow needs a Gitea-detection no-op or a Gitea REST replacement.
• ❌ Harness Replays / Harness Replays — DinD bind-mount of tests/harness/cf-proxy/nginx.conf fails because act_runner doesn't expose the workspace path to nested Docker. Chronic on every staging workspace-server commit.
• ❌ Handlers Postgres Integration — IPv6 [::1]:5432 resolution race against the IPv4-only service container. Intermittent: this same PR's first commit (16868c4e) passed HPI; the second commit (b6646910) hit the flake. One-char fix (127.0.0.1).

Filed #88 with proposed fixes and recommended order. Keeping those out of this PR per feedback_gitea_actions_migration_audit_pattern (bundle per-repo, not per-finding).

Why the 2nd commit

Folded a separate root-cause fix for TestPooledWithEICTunnel_PanicPoisonsEntry (eic_tunnel_pool.go data race) into this PR — that race was the actual reason CI / Platform (Go) was intermittently red across staging. Capturing poolJanitorInterval at pool construction stops the janitor goroutine from racing with t.Cleanup-driven swaps of the package var. Local go test -race ./... is now fully green; this run confirms it lands green on CI too.

Ready for review

• Code: plugins_install_eic.go (new) + dispatch wired in plugins_install_pipeline.go + plugins_install.go + plugins.go + router.go.
• Tests: plugins_install_eic_test.go covers the 4 dispatch shapes (SaaS happy path, EIC error → 502, no-instance → 503, lookup-error → 503) + symmetric uninstall + pure-fn shell shape + tar.gz round-trip.
• Docs: docs/architecture/backends.md plugins row flipped 🔴 → ✅ with a one-line description of the new dispatch.
• Race fix: eic_tunnel_pool.go capture-at-construction (4-line struct+constructor change + 1-line janitor read swap).

## CI verdict Functional checks all green: **CI / Platform (Go), Handlers-Postgres-Integration (on the 1st commit), E2E API Smoke, Playwright, CodeQL, Canvas, Python, Runtime PR-Built, Secret-scan**. Total **23/26 success**. The 3 reds are pre-existing Gitea-vs-GitHub Actions infrastructure issues — none from this PR's code path. Verified by reproducing each failure on `origin/staging` directly: - ❌ `pr-guards / disable-auto-merge-on-push` — `gh` CLI calls GitHub's GraphQL → Gitea returns 405. Fails on **every** Gitea PR push since the migration. Workflow needs a Gitea-detection no-op or a Gitea REST replacement. - ❌ `Harness Replays / Harness Replays` — DinD bind-mount of `tests/harness/cf-proxy/nginx.conf` fails because act_runner doesn't expose the workspace path to nested Docker. Chronic on every staging workspace-server commit. - ❌ `Handlers Postgres Integration` — IPv6 `[::1]:5432` resolution race against the IPv4-only service container. Intermittent: this same PR's first commit (`16868c4e`) passed HPI; the second commit (`b6646910`) hit the flake. One-char fix (`127.0.0.1`). Filed #88 with proposed fixes and recommended order. Keeping those out of this PR per `feedback_gitea_actions_migration_audit_pattern` (bundle per-repo, not per-finding). ## Why the 2nd commit Folded a separate root-cause fix for `TestPooledWithEICTunnel_PanicPoisonsEntry` (`eic_tunnel_pool.go` data race) into this PR — that race was the actual reason `CI / Platform (Go)` was intermittently red across staging. Capturing `poolJanitorInterval` at pool construction stops the janitor goroutine from racing with `t.Cleanup`-driven swaps of the package var. Local `go test -race ./...` is now fully green; this run confirms it lands green on CI too. ## Ready for review - Code: `plugins_install_eic.go` (new) + dispatch wired in `plugins_install_pipeline.go` + `plugins_install.go` + `plugins.go` + `router.go`. - Tests: `plugins_install_eic_test.go` covers the 4 dispatch shapes (SaaS happy path, EIC error → 502, no-instance → 503, lookup-error → 503) + symmetric uninstall + pure-fn shell shape + tar.gz round-trip. - Docs: `docs/architecture/backends.md` plugins row flipped 🔴 → ✅ with a one-line description of the new dispatch. - Race fix: `eic_tunnel_pool.go` capture-at-construction (4-line struct+constructor change + 1-line janitor read swap).

claude-ceo-assistant referenced this issue from a commit 2026-05-08 00:06:10 +00:00

fix(ci): close 3 chronic Gitea-Actions workflow flakes (closes #88)

claude-ceo-assistant referenced this pull request 2026-05-08 00:06:32 +00:00

fix(ci): close 3 chronic Gitea-Actions workflow flakes (closes #88) #92

claude-ceo-assistant added 1 commit 2026-05-08 00:21:16 +00:00

Merge branch 'staging' into fix/saas-plugin-install-eic 7f61206a18

claude-ceo-assistant added 1 commit 2026-05-08 01:29:15 +00:00

Merge branch 'staging' into fix/saas-plugin-install-eic 576166c8c3

claude-ceo-assistant added 1 commit 2026-05-08 02:03:59 +00:00

Merge branch 'staging' into fix/saas-plugin-install-eic d201b13b93

Ghost approved these changes 2026-05-08 02:15:49 +00:00

Ghost left a comment

First-time contributor

Copy Link

SaaS (EC2-per-workspace) plugin install via EIC SSH. Mirrors Files API #1702 pattern. 4-tier dispatch ladder (local docker → EIC SSH → external runtime hint → 503). Comprehensive test coverage. By claude-ceo-assistant. Approved.

SaaS (EC2-per-workspace) plugin install via EIC SSH. Mirrors Files API #1702 pattern. 4-tier dispatch ladder (local docker → EIC SSH → external runtime hint → 503). Comprehensive test coverage. By claude-ceo-assistant. Approved.

claude-ceo-assistant merged commit b11044f885 into staging 2026-05-08 02:15:50 +00:00

claude-ceo-assistant referenced this issue from a commit 2026-05-08 02:15:50 +00:00

fix(plugins): SaaS (EC2-per-workspace) install/uninstall via EIC SSH (#84)

Ghost referenced this issue from a commit 2026-05-08 02:19:01 +00:00

fix(ci): port 3 verified-green CI fixes from staging to main

claude-ceo-assistant referenced this pull request 2026-05-08 02:29:08 +00:00

fix(ci): port 3 verified-green CI fixes from staging to main (cherry-pick) #101

claude-ceo-assistant referenced this issue from a commit 2026-05-08 02:46:41 +00:00

chore: promote accumulated staging fixes to main (vitest, postgres, e2e-api, eic, plugins) (#99)

Sign in to join this conversation.

Reviewers

No reviewers

Ghost

Labels

Clear labels   

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

No Label

tier:high

tier:low

tier:medium

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

3 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#84

No description provided.