Root cause:
Dockerfile.tenant chowns /canvas /platform /memory-plugin /migrations
to canvas:canvas (line ~119) but not /org-templates. The image is
built as root, COPY-ed templates inherit root:root 0755. The platform
binary then runs as the canvas user (uid 1000) because of the USER
directive on line ~124, so when the !external resolver
(org_external.go, internal#77 / task #222) tries
os.MkdirAll("/org-templates/<tmpl>/.external-cache/<repo>") on first
import, mkdir(2) returns EACCES and the import handler returns 400
"org template expansion failed" (org.go:592). The user-facing error
is generic; only the server log carries:
Org import: refusing import: !include expansion failed:
!external at line 156: fetch git.moleculesai.app/molecule-ai/molecule-dev-department@v1.0.0:
mkdir cache root: mkdir /org-templates/molecule-dev/.external-cache: permission denied
Repro:
Tenant staging-cplead-2 (canary AWS 004947743811, image SHA
a93c4ce17725...). POST /org/import {"dir":"molecule-dev"} returns 400
while POST /org/import {"dir":"free-beats-all"} returns 201 — only
templates with !external trip the bug.
Fix:
Add /org-templates to the chown -R argv. One-line change. Same
ownership shape as the other writable platform-state dirs.
Why this is safe for prod:
* The platform binary already needs read access to /org-templates,
so canvas:canvas owning it doesn't widen any attack surface.
* /org-templates is image-resident, not bind-mounted; chown applies
inside the image layers and prod tenants get the fix on next
image rebuild + redeploy. Live prod tenants are unaffected until
the next deploy (no orgs currently using !external in prod —
molecule-dev consumers are all internal staging).
Verification:
After hand-applying the chown live (docker exec --user 0 ... chown -R
canvas:canvas /org-templates/molecule-dev), POST /org/import
{"dir":"molecule-dev"} returns 201 with 39 workspaces; cp-lead +
CP-BE + CP-QA + CP-Security all reach status=online within ~2 min.
Refs:
internal#77 — !external RFC (Phase 3a)
task #222 — resolver PR (introduced the unflagged-permission
dependency this fixes)
Live incident 2026-05-10 — staging-cplead-2 import failed,
chown-on-host workaround in place pending image rebuild
Issue #212: POST /workspaces with runtime=external and a URL wrote the
URL directly to the DB without validateAgentURL checking (the same check
that registry.go:324 applies to the heartbeat path). An attacker with
AdminAuth could register a workspace URL at a cloud metadata endpoint
(169.254.169.254) and exfiltrate IAM credentials when the platform
fires pre-restart drain signals.
Changes:
- workspace.go: add validateAgentURL(payload.URL) guard before the
UPDATE at line 386. 400 on unsafe URL, no DB write occurs.
- workspace_test.go: add 3 regression tests:
- TestWorkspaceCreate_ExternalURL_SSRFSafe: safe public URL → 201
- TestWorkspaceCreate_ExternalURL_SSRFMetadataBlocked: 169.254.169.254 → 400
- TestWorkspaceCreate_ExternalURL_SSRFLoopbackBlocked: 127.0.0.1 → 400
Both unsafe tests assert zero DB calls (the handler rejects before
any transaction).
Ref: issue #212.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Issue #214: documents the MOLECULE_ENV=production requirement for
staging/prod tenants to lock the /admin/workspaces/:id/test-token route.
Also adds a startup INFO log in main.go when the route is enabled, so
operators can confirm the setting in boot logs without having to probe
the endpoint directly.
Ref: issue #214.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a 4th fallback step to the token chain (cache > API > env > static)
so workspace git/gh operations survive a platform outage without requiring
a restart or platform-side fix. Addresses the 2026-05-08 incident where
every workspace lost git+gh auth simultaneously when the
/github-installation-token endpoint returned 500.
Operator places a PAT in ${CONFIGS_DIR:-/configs}/.github-token
(no root needed — /configs is agent-writable). Both _fetch_token
(git credential helper path) and _refresh_gh (gh CLI daemon path)
gain the static fallback so git and gh both recover post-incident.
Pure additive — existing cache > API > env chain is unchanged.
Empty static file is rejected (whitespace-stripped before use).
Static path never writes the cache, so the API recovers transparently
on the next refresh cycle when it comes back online.
Ref: issue #140.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
StatusBadge: all 3 status variants, aria-label, role=status, config class names.
ValidationHint: error/valid/neutral states, warning icon, valid icon, class names.
Spinner: sm/md/lg size classes, aria-hidden, motion-safe:animate-spin.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of issue #213: canary-verify.yml still used GHCR
(ghcr.io/molecule-ai/platform-tenant) while
publish-workspace-server-image.yml migrated to ECR on 2026-05-07
(commit 10e510f5). Canary smoke tests were silently testing a stale
GHCR image while actual staging/prod tenants ran the ECR build.
The POST /org/import and POST /workspaces routes were missing from
the ECR binary (likely a Docker layer-caching artefact during the
staging push window) but smoke tests passed because they never tested
the ECR image at all.
Changes:
- canary-verify.yml: migrate promote-to-latest from GHCR crane tag
ops to the CP redeploy-fleet endpoint (same mechanism as
redeploy-tenants-on-main.yml). The wait-for-canaries step already
read SHA from the running tenant /health (registry-agnostic), so
no change needed there. Pre-fix promote step used `crane tag` against
GHCR, which was never updated after the ECR migration.
- redeploy-tenants-on-main.yml: update stale comments that reference
GHCR to reflect ECR; replace the 30s GHCR CDN propagation wait
with a no-op comment (ECR has no CDN cache to wait for).
- scripts/canary-smoke.sh: add POST /org/import and POST /workspaces
smoke tests (steps 6-8). These assert HTTP 401 unauthenticated
(proves AdminAuth enforced AND the route is compiled in — 404 would
mean route missing from binary). GET /workspaces was already covered;
POST was the untested gap.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
publish-runtime.yml was dead on Gitea Actions because Gitea reads
.gitea/workflows/, not .github/workflows/ (the GitHub Actions paths are
ignored). Issue #206 identified this as one of three bugs blocking the
runtime versioning pipeline.
Changes:
- Add .gitea/workflows/publish-runtime.yml (canonical Gitea version)
- Drop environment: + id-token: write (Gitea has no OIDC/OAuth)
- Replace pypa/gh-action-pypi-publish with twine upload using PYPI_TOKEN secret
- Replace github.ref_name with ${GITHUB_REF#refs/tags/} (Gitea exposes github.ref)
- Drop merge_group trigger (Gitea has no merge queue)
- Drop staging branch trigger (staging branch does not exist)
- Cascade step unchanged (DISPATCH_TOKEN + Gitea API already compatible)
- Add DEPRECATED notice to .github/workflows/publish-runtime.yml
Required secrets (repo Settings → Actions → Variables and Secrets):
PYPI_TOKEN: PyPI API token for molecule-ai-workspace-runtime
DISPATCH_TOKEN: Gitea PAT with write:repo on template repos (already used)
Closes#206 (publish-runtime Gitea port).
dorny/paths-filter is GitHub-Actions-only and does not work correctly on
Gitea Actions — it silently returns no file changes regardless of what
files were modified, causing the harness-replays workflow to silently
skip on Gitea even when workspace-server/** or canvas/** files change.
Verified: zero harness-replays statuses on PR #188 and #168 (both changed
workspace-server files) vs GitHub Actions where the same workflow
correctly detects changes.
Replace with a shell-based approach that uses:
- github.event.pull_request.base.sha (Gitea + GitHub: merge-base for PRs)
- github.event.before (Gitea + GitHub: previous tip for pushes)
- git diff --name-only <BASE> github.sha (portable git, works on both platforms)
Also adds detect-changes.debug output so future no-op passes show WHY
the workflow decided to skip, and the first real run on Gitea will
confirm the diff detection is working.
Closes#141 (followup: root-cause fix still TBD — failure logs
inaccessible via Gitea Actions API).
