When the workspace EC2's sshd is unresponsive (mid-restart, SG drop,
AMI without ec2-instance-connect), the canvas's xterm shows the user's
typed bytes echoed back by the workspace-server's *local* PTY (cooked +
echo mode before ssh sets it raw post-handshake) and then closes
silently when Cloudflare's idle WebSocket timer fires (~100s) — with no
"Connection refused" or "Permission denied" output ever reaching the
user. This is what hongmingwang's hermes terminal looked like 2026-04-30
right after the heartbeat-fix redeploy: status="online" but the shell
appeared dead.
Caught reproducibly by holding a fresh /workspaces/<id>/terminal
WebSocket open for 60s — server sent zero frames except the local-PTY
echo of one keystroke typed at t=8s. ssh was hung at handshake; bash
never saw the byte.
Fix: add `-o ConnectTimeout=10` to ssh args. Now the failure surfaces
as a real ssh error message in the terminal within 10s, instead of
masquerading as a silently dead shell over the next ~100s. Doesn't
diagnose *why* sshd isn't responding (separate investigation), but
it does mean the user gets actionable feedback within seconds.
Behavior-based regression test asserts `-o ConnectTimeout=N` is in the
ssh argv — pins presence, not the literal value, so operators can tune
without breaking the gate. Verified to FAIL on pre-fix code (matched
the literal arg pair) and PASS on fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #1885 introduced a regression: HandleConnect called wsauth.ValidateToken
for any bearer token when X-Workspace-ID ≠ workspaceID. Org-scoped tokens
(org_api_tokens table) are not in workspace_auth_tokens, so ValidateToken
always returned ErrInvalidToken for them → hard 401 for all A2A routing
that uses org tokens.
Fix: if WorkspaceAuth already validated an org token (org_token_id set in
gin context by orgtoken.Validate), skip the workspace_auth_tokens lookup and
trust the X-Workspace-ID claim. Hierarchy enforcement via canCommunicateCheck
is unchanged — org token holders are still subject to the workspace hierarchy.
Workspace-scoped tokens continue to require ValidateToken binding. Invalid
tokens (neither workspace-bound nor org-level) still return 401. This closes
the regression while preserving the KI-005 security property.
Add TestKI005_OrgToken_SkipsValidateToken to terminal_test.go as a regression
guard for this exact path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Brings the staging branch up to date with main's feature-fix stream so
every staging-targeted PR stops tripping on pre-existing rot. Before
this merge, staging had 30+ compile + test failures from fix PRs that
landed on main but never reached staging — primarily #1755's panic-
cascade + schema-drift alignments.
After this merge the handlers package goes from 30+ fails → 2 pre-
existing nil-docker test panics (TestCopyFilesToContainer_CWE22_
RejectsTraversal + TestDeleteViaEphemeral_F1085_RejectsTraversal),
both authored on staging and broken before this promotion. Tracked
separately; not a merge regression.
## Conflicts resolved
1. docs/marketing/campaigns/discord-adapter-announcement/announcement.md
— deleted on main (9d0d213: "move sensitive strategy + research to
internal repo"), modified on staging. Deletion wins: marketing
content moved out of the public monorepo per that commit's intent.
The content lives in the internal repo.
2. workspace-server/internal/handlers/container_files.go — staging's
rmTarget version kept. Main's version had `Cmd: []string{"rm",
"-rf", "/configs/" + filePath}` which concatenates raw filePath
AFTER the prefix-check on rmTarget, defeating the path-traversal
guard (a "../etc/passwd" input passes validation but the rm cmd
then traverses). Staging's `Cmd: []string{"rm", "-rf", rmTarget}`
uses the validated path. Keeping staging's more-secure variant.
## Includes build unblockers from #1769 / #1782
- terminal.go: malformed handleLocalConnect repaired
- terminal_test.go: missing braces in TestHandleConnect_RoutesToLocal
- workspace_crud.go: unused imports + duplicate strField block
- container_files_test.go: duplicate contains() removed (uses the one
in workspace_provision_test.go, same package)
## Verification
- go build ./... ✅ clean
- go vet ./... ✅ clean
- go test -race ./... — 18/20 packages green; 2 test panics in
internal/handlers are pre-existing on staging (documented above)
Two latent bugs the self-hosted Mac mini had been hiding. Both caught
by the newer toolchain on ubuntu-latest runners after PR #1626.
1. workspace-server/internal/handlers/terminal.go:442
`fmt.Sprintf("%s:%d", host, port)` flagged by go vet as unsafe
for IPv6 (it omits the required [::] brackets). Replaced with
`net.JoinHostPort(host, strconv.Itoa(port))` which handles both
IPv4 and IPv6 correctly. No runtime behaviour change — the only
call site passes "127.0.0.1", so the bug would never trigger in
practice, but vet is right to flag it as a latent correctness
issue.
2. workspace/tests/test_a2a_executor.py::test_set_current_task_updates_heartbeat
`MagicMock()` auto-creates attributes on first access, so
`getattr(heartbeat, "active_tasks", 0)` in shared_runtime.py
returned a MagicMock rather than the default 0. Adding 1 to a
MagicMock returns another MagicMock, so the assertion
`heartbeat.active_tasks == 1` never held. Seeding
`heartbeat.active_tasks = 0` before the first call makes
getattr() return a real int, matching how the real HeartbeatLoop
class initialises itself.
Both pre-existed on main and were hidden by the older Python / Go
toolchains on the Mac mini runner. Verified locally (venv pytest
pass, `go vet ./...` + `go build ./...` clean on workspace-server).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(lint): unblock Platform Go CI — suppress 8 pre-existing errcheck warnings
golangci-lint errcheck has been flagging these since before this PR —
not regressions from the restart fix, just long-standing debt that
blocks Platform (Go) CI from ever going green. Prefix ignored returns
with `_ =` to make the signal explicit without changing behavior:
- channels/lark_test.go:97 (w.Write) + :118 (resp.Body.Close)
- channels/channels_test.go:620 + :760 (mockDB.Close in t.Cleanup)
- channels/manager.go:131 + :196 (defer rows.Close via closure wrapper)
- channels/manager.go:206–207 (json.Unmarshal into struct fields)
- artifacts/client_test.go:195, 237, 297 (json.Decode in test handlers)
The manager.go defer patch uses `defer func() { _ = rows.Close() }()`
since errcheck doesn't allow the `_ =` prefix directly on `defer`.
Build + `go test ./...` green locally for internal/channels and
internal/artifacts. The manager.go change touches production code so
I re-ran the channels test suite; passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: trigger PR refresh
* test(handlers): add CWE-22 regression suite + KI-005 terminal access fix + tests
container_files_test.go (152 lines):
- 11 path-traversal test cases for copyFilesToContainer (F1501/CWE-22)
- Tests nil Docker client — validation logic runs before any Docker call
terminal.go KI-005 security fix (backport from ship/security-fix 6de7530c):
- Enforce CanCommunicate hierarchy check before granting terminal access
- Shell access is more dangerous than A2A message-passing; apply the
same hierarchy check used by A2A and discovery endpoints
- When X-Workspace-ID header is present and bearer token is valid
(ValidateAnyToken), reject unless CanCommunicate(callerID, targetID)
- Canvas/molecli callers without X-Workspace-ID header pass through to
WorkspaceAuth middleware for existing bearer check
- canCommunicateCheck exposed as package var for testability
terminal_test.go (5 test cases):
- TestTerminalConnect_KI005_RejectsUnauthorizedCrossWorkspace
- TestTerminalConnect_KI005_AllowsOwnTerminal
- TestTerminalConnect_KI005_SkipsCheckWithoutHeader
- TestTerminalConnect_KI005_RejectsInvalidToken
- TestTerminalConnect_KI005_AllowsSiblingWorkspace
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Molecule AI Core-BE <core-be@agents.moleculesai.app>
Proven by end-to-end testing against a live Hermes workspace EC2:
CP-provisioned workspaces run the agent as a NATIVE process under
the ubuntu user, not inside a Docker container. The earlier
\`aws ec2-instance-connect ssh -- docker exec -it ws-X bash\` was
doubly wrong:
- aws-cli's \`ssh\` subcommand doesn't accept a trailing command
- Even if it did, there's no container to exec into
Replaced with a three-step pipeline that matches what actually
works when run by hand:
1. ssh-keygen — ephemeral ed25519 per session
2. aws ec2-instance-connect send-ssh-public-key --instance-os-user ubuntu
3. aws ec2-instance-connect open-tunnel --local-port N (runs in background)
4. ssh -p N -i <key> ubuntu@127.0.0.1
Infra prerequisites (verified in docs/infra/workspace-terminal.md):
- EIC service-linked role created
- EIC Endpoint in the workspace VPC (we created eice-08b035ec8789202f9)
- Workspace SG allows 22/tcp from the EIC Endpoint's SG
- molecule-cp IAM: ec2:DescribeInstances + ec2-instance-connect:*
Changes in this commit:
- eicSSHOptions struct carries session inputs between factories
- openTunnelCmd + sshCommandCmd + sendSSHPublicKey are package vars
so tests can stub them individually
- Default OS user is \"ubuntu\" (Ubuntu 24.04 CP AMI). Override via
WORKSPACE_EC2_OS_USER env var if the AMI changes
- AWS_REGION env var respected; default us-east-2 matches current CP
- pickFreePort + waitForPort helpers — no hardcoded ports, tolerates
multiple concurrent sessions
- Tests updated: two argv-shape regressions for open-tunnel + ssh
(SSH shape was the silent-drift case that caused the first failure)
Refs: #1528, #1531
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the last CP-provisioned-workspace gap: Terminal tab now works
for workspaces running on separate EC2 instances. Follow-up to
#1531 which added instance_id persistence.
How it works:
- HandleConnect checks workspaces.instance_id
- Empty → existing local Docker path (unchanged)
- Set → spawn `aws ec2-instance-connect ssh --connection-type eice
--instance-id X --os-user ec2-user -- docker exec -it ws-Y
/bin/bash` under creack/pty, bridge pty ↔ canvas WebSocket
Why subprocess AWS CLI instead of native AWS SDK:
- EIC Endpoint tunnel needs a signed WebSocket with specific framing
- aws-cli v2 implements it correctly; reimplementing in Go is ~500
lines of crypto + WS protocol work for zero user-visible benefit
- Tenant image picks up 1MB of aws-cli + openssh-client via apk
Handler design:
- sshCommandFactory is a var so tests can stub it (no real aws calls)
- Context cancellation propagates both ways (WS close → kill ssh;
ssh exit → close WS)
- User-visible error points at docs/infra/workspace-terminal.md when
EIC wiring is incomplete (common bootstrap failure)
Tests:
- TestHandleConnect_RoutesToRemote — instance_id in DB → CP branch
- TestHandleConnect_RoutesToLocal — empty instance_id → local branch
- TestSshCommandFactory_BuildsEICCommand — argv shape regression guard
Dockerfile.tenant: + openssh-client + aws-cli (Alpine main repo)
Refs: #1528, #1531
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
