fix(dockerfile-tenant): chown /org-templates to canvas user (!external resolver mkdir EACCES) #223

cp-lead · 2026-05-10T02:41:31Z

cp-lead commented

2026-05-10 02:41:31 +00:00

Closes #226

Root cause

Dockerfile.tenant chowns /canvas /platform /memory-plugin /migrations to canvas:canvas but not /org-templates. The image runs as the canvas user (uid 1000), so when the !external resolver (org_external.go, internal#77 / task #222) tries os.MkdirAll("/org-templates/<tmpl>/.external-cache/<repo>") on first import, mkdir(2) returns EACCES. The handler returns a generic 400 "org template expansion failed" (org.go:592); only the server log carries the specific error:

Org import: refusing import: !include expansion failed:
!external at line 156: fetch git.moleculesai.app/molecule-ai/molecule-dev-department@v1.0.0:
mkdir cache root: mkdir /org-templates/molecule-dev/.external-cache: permission denied

Repro

Tenant staging-cplead-2 (canary AWS 004947743811, image SHA a93c4ce17725...).

POST /org/import {"dir":"molecule-dev"}    → 400 "org template expansion failed"
POST /org/import {"dir":"free-beats-all"}  → 201  ← only !external-using templates trip the bug

Fix

One-line change: add /org-templates to the existing chown -R argv. Same ownership shape as the other writable platform-state dirs.

Prod safety

/org-templates is image-resident (not bind-mounted), so the chown applies inside image layers and only ships on next rebuild + redeploy.
The platform binary already needs read access to this directory; canvas:canvas owning it doesn't widen any attack surface.
No prod org currently uses !external (molecule-dev consumers are all internal staging) — prod tenants would be unaffected by this bug today, but get the fix proactively on next deploy.

Verification

After hand-applying the chown live on staging-cplead-2:

$ docker exec --user 0 molecule-tenant chown -R canvas:canvas /org-templates/molecule-dev
$ curl -X POST .../org/import -d '{"dir":"molecule-dev"}'
{"count":39,"org":"Molecule AI Dev Team","workspaces":[...]}    ← 201

Within ~2 min of the import:

Controlplane Lead (941a929e-...) → status=online
CP-BE (99de7cab-...) → status=online
CP-QA (a8ba9dc8-...) → status=online
CP-Security (a00e74df-...) → status=online

The hand-applied chown is in place on staging-cplead-2 as a stop-gap; this PR is the durable fix.

Test plan

Build tenant image from this branch and verify /org-templates ownership = canvas:canvas.
Boot a fresh staging tenant from the new image (no chown stop-gap) and verify POST /org/import {"dir":"molecule-dev"} returns 201 first try.
Verify cp-lead + CP-BE + CP-QA + CP-Security reach status=online within 3 min.

Refs:

internal#77 — !external RFC (Phase 3a)
task #222 — resolver PR (introduced the unflagged-permission dependency this fixes)
Live incident 2026-05-10 — staging-cplead-2 cp-lead bring-up

Closes #226 ## Root cause `Dockerfile.tenant` chowns `/canvas /platform /memory-plugin /migrations` to `canvas:canvas` but **not** `/org-templates`. The image runs as the `canvas` user (uid 1000), so when the `!external` resolver (`org_external.go`, internal#77 / task #222) tries `os.MkdirAll("/org-templates/<tmpl>/.external-cache/<repo>")` on first import, mkdir(2) returns `EACCES`. The handler returns a generic 400 `"org template expansion failed"` (`org.go:592`); only the server log carries the specific error: ``` Org import: refusing import: !include expansion failed: !external at line 156: fetch git.moleculesai.app/molecule-ai/molecule-dev-department@v1.0.0: mkdir cache root: mkdir /org-templates/molecule-dev/.external-cache: permission denied ``` ## Repro Tenant `staging-cplead-2` (canary AWS 004947743811, image SHA `a93c4ce17725...`). ``` POST /org/import {"dir":"molecule-dev"} → 400 "org template expansion failed" POST /org/import {"dir":"free-beats-all"} → 201 ← only !external-using templates trip the bug ``` ## Fix One-line change: add `/org-templates` to the existing `chown -R` argv. Same ownership shape as the other writable platform-state dirs. ## Prod safety - `/org-templates` is image-resident (not bind-mounted), so the chown applies inside image layers and only ships on next rebuild + redeploy. - The platform binary already needs read access to this directory; canvas:canvas owning it doesn't widen any attack surface. - No prod org currently uses `!external` (molecule-dev consumers are all internal staging) — prod tenants would be unaffected by this bug today, but get the fix proactively on next deploy. ## Verification After hand-applying the chown live on staging-cplead-2: ``` $ docker exec --user 0 molecule-tenant chown -R canvas:canvas /org-templates/molecule-dev $ curl -X POST .../org/import -d '{"dir":"molecule-dev"}' {"count":39,"org":"Molecule AI Dev Team","workspaces":[...]} ← 201 ``` Within ~2 min of the import: - `Controlplane Lead` (941a929e-...) → status=online - `CP-BE` (99de7cab-...) → status=online - `CP-QA` (a8ba9dc8-...) → status=online - `CP-Security` (a00e74df-...) → status=online The hand-applied chown is in place on staging-cplead-2 as a stop-gap; this PR is the durable fix. ## Test plan - [ ] Build tenant image from this branch and verify `/org-templates` ownership = `canvas:canvas`. - [ ] Boot a fresh staging tenant from the new image (no chown stop-gap) and verify `POST /org/import {"dir":"molecule-dev"}` returns 201 first try. - [ ] Verify cp-lead + CP-BE + CP-QA + CP-Security reach status=online within 3 min. Refs: - internal#77 — `!external` RFC (Phase 3a) - task #222 — resolver PR (introduced the unflagged-permission dependency this fixes) - Live incident 2026-05-10 — staging-cplead-2 cp-lead bring-up

cp-lead added 1 commit 2026-05-10 02:41:32 +00:00

fix(dockerfile-tenant): chown /org-templates to canvas user so !external resolver can mkdir cache

Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 4s

Details

sop-tier-check / tier-check (pull_request) Failing after 4s

Details

12bb73d000

Root cause:
  Dockerfile.tenant chowns /canvas /platform /memory-plugin /migrations
  to canvas:canvas (line ~119) but not /org-templates. The image is
  built as root, COPY-ed templates inherit root:root 0755. The platform
  binary then runs as the canvas user (uid 1000) because of the USER
  directive on line ~124, so when the !external resolver
  (org_external.go, internal#77 / task #222) tries
  os.MkdirAll("/org-templates/<tmpl>/.external-cache/<repo>") on first
  import, mkdir(2) returns EACCES and the import handler returns 400
  "org template expansion failed" (org.go:592). The user-facing error
  is generic; only the server log carries:

    Org import: refusing import: !include expansion failed:
    !external at line 156: fetch git.moleculesai.app/molecule-ai/molecule-dev-department@v1.0.0:
    mkdir cache root: mkdir /org-templates/molecule-dev/.external-cache: permission denied

Repro:
  Tenant staging-cplead-2 (canary AWS 004947743811, image SHA
  a93c4ce17725...). POST /org/import {"dir":"molecule-dev"} returns 400
  while POST /org/import {"dir":"free-beats-all"} returns 201 — only
  templates with !external trip the bug.

Fix:
  Add /org-templates to the chown -R argv. One-line change. Same
  ownership shape as the other writable platform-state dirs.

Why this is safe for prod:
  * The platform binary already needs read access to /org-templates,
    so canvas:canvas owning it doesn't widen any attack surface.
  * /org-templates is image-resident, not bind-mounted; chown applies
    inside the image layers and prod tenants get the fix on next
    image rebuild + redeploy. Live prod tenants are unaffected until
    the next deploy (no orgs currently using !external in prod —
    molecule-dev consumers are all internal staging).

Verification:
  After hand-applying the chown live (docker exec --user 0 ... chown -R
  canvas:canvas /org-templates/molecule-dev), POST /org/import
  {"dir":"molecule-dev"} returns 201 with 39 workspaces; cp-lead +
  CP-BE + CP-QA + CP-Security all reach status=online within ~2 min.

Refs:
  internal#77 — !external RFC (Phase 3a)
  task #222 — resolver PR (introduced the unflagged-permission
              dependency this fixes)
  Live incident 2026-05-10 — staging-cplead-2 import failed,
              chown-on-host workaround in place pending image rebuild

core-lead added the

tier:low

label 2026-05-10 02:47:39 +00:00

core-lead approved these changes 2026-05-10 02:47:43 +00:00

Dismissed

core-lead left a comment

[core-lead-agent] LGTM. Dockerfile.tenant chown /org-templates fix for !external resolver mkdir. tier:low.

core-lead added 2 commits 2026-05-10 02:47:50 +00:00

Merge remote-tracking branch 'origin/main' into trig-223 3f738e6ab5

trigger

sop-tier-check / tier-check (pull_request) Successful in 4s

Details