feat(org-import): inject per-role persona env from operator-host bootstrap dir #110

Merged

claude-ceo-assistant merged 14 commits from feat/persona-env-injection into main 2026-05-08 15:17:17 +00:00

Conversation 0 Commits 14 Files Changed 10 +1913 -2

14 Commits

Author	SHA1	Message	Date
claude-ceo-assistant	43b33bcaa5	feat(org-import): inject per-role persona env from operator-host bootstrap dir ... Wires the 28 dev-tree persona credentials minted 2026-05-08 into the workspace-secrets path used by org_import. When a workspace.yaml carries `role: <name>`, the importer now reads $MOLECULE_PERSONA_ROOT/<role>/env (default /etc/molecule-bootstrap/personas/<role>/env, populated by the bootstrap kit on the tenant host) and merges the role's GITEA_USER / GITEA_TOKEN / GITEA_TOKEN_SCOPES / GITEA_USER_EMAIL / GITEA_SSH_KEY_PATH into the same envVars map that already feeds workspace_secrets via parseEnvFile + crypto.Encrypt + INSERT. PRECEDENCE Persona env is the LOWEST layer: 0. Persona env (per-role) 1. Org root .env (shared) 2. Workspace .env (per-workspace) Each later layer overrides the previous, so a workspace .env can pin a different GITEA_TOKEN if it ever needs to (testing, override). WHY THIS LAYERING Workspaces should boot with the role's identity by default. .env files stay the explicit-override mechanism for the (rare) case where a workspace needs to deviate. No new behavior for workspaces with no role: persona load is silent no-op when ws.Role is empty or unsafe. SECURITY isSafeRoleName accepts only [A-Za-z0-9_-]+ (no '..', '/', or separators) — admin-only construct, but defense-in-depth keeps the persona dir shape invariant. Test TestLoadPersonaEnvFile_RejectsTraversal pins the rejection set against a planted target file. OPERATOR-HOST CONTRACT The 28 persona env files live at /etc/molecule-bootstrap/personas/<role>/env (mode 600, owner root:root) with the per-role token-scope tailoring Hongming approved 2026-05-08 (D5). Synced via task #241. Override via MOLECULE_PERSONA_ROOT for tests + non-prod hosts. TESTS (7 new, all green) TestLoadPersonaEnvFile_HappyPath — typical persona-env shape TestLoadPersonaEnvFile_MissingDir — silent no-op when file absent TestLoadPersonaEnvFile_EmptyRole — silent no-op when role empty TestLoadPersonaEnvFile_RejectsTraversal — planted file unreachable via '../../etc/passwd' etc. TestLoadPersonaEnvFile_DefaultRoot — falls back to /etc/... TestLoadPersonaEnvFile_OverwritesEmptyMap TestIsSafeRoleName_Acceptance — positive + negative role names PHASE 4 SELF-REVIEW (FIVE-AXIS) Correctness: No finding — additive change, silent no-op on the ws.Role=='' path covers every existing workspace; tests cover happy path + each rejection mode + missing-dir. Readability: No finding — helper sits next to parseEnvFile in org_helpers.go with a comment block explaining WHY persona is lowest precedence. Architecture: No finding — fits the existing 'merge .env into envVars then INSERT INTO workspace_secrets' pattern that's been in place since the .env-driven workspace secrets feature; no new dependencies, no new tables. Security: Required (addressed) — path traversal blocked by isSafeRoleName. No finding beyond that since persona files are admin-managed and the helper does not log token values. Performance: No finding — one extra os.ReadFile per workspace at import time; amortized over workspace lifetime, cost is negligible. REFS internal#85 — RFC for SOP Phase 4 + structured Five-Axis (parent context) Saved memories: feedback_per_agent_gitea_identity_default, feedback_unified_credentials_file Task #241 — operator-host sync (already DONE; populated 28 dirs) Task #242 — this PR Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-08 07:09:40 -07:00
devops-engineer	7d3a6a46c5	chore: sync main → staging (auto, ae2d9eab)	2026-05-08 13:30:46 +00:00
claude-ceo-assistant	5abc4f74ca	harden(org-external): token via http.extraHeader, .complete cache marker, ref .. deny, naming cleanup (#107) ... Five-Axis self-review pass on the !external resolver work (PRs #105+#106) caught three real issues that the unstructured 3-weakest review missed: 1. Cache validity gap — partial cache writes looked complete 2. Token persistence — token in URL userinfo got persisted to .git/config 3. Misleading function name post-refactor This PR fixes all three: - .complete marker file written atomically; wipe-and-refetch on partial cache - Token via -c http.extraHeader, never embedded in URL - Defense-in-depth ref .. deny (was already validated by repoSafeRefRegex but explicit + tested) - Renamed buildCloneURL -> buildExternalCloneURL (collision with artifacts.go), rewriteFilesDirAndIncludes -> rewriteFilesDir - Removed unused redactToken/shortHash helpers and crypto/sha1, encoding/hex, fmt imports Approved by platform-engineer 2026-05-08T12:55Z.	2026-05-08 13:04:00 +00:00
claude-ceo-assistant	c72d0a5383	harden(org-external): token via http.extraHeader, .complete cache marker, ref '..' deny, naming cleanup ... Self-review of molecule-core PR #105 + #106 (the !external resolver chain) surfaced 3 real correctness/security gaps and 2 readability nits. Fixes all four in one PR since they're the same file's hardening. (1) TOKEN LEAKAGE — fixed Before: gitFetcher built clone URLs with auth in userinfo (https://oauth2:TOKEN@host/repo.git). Two leak paths: a. Token persisted in cloned repo's .git/config b. Token could appear in clone error output captured via cmd.CombinedOutput() After: clone URL has no userinfo (https://host/repo.git). Auth is layered on via -c http.extraHeader=Authorization: token ... which sends the header per-request without persisting. Plus a redactToken() pass over any error string before it surfaces in fmt.Errorf, as belt-and-braces. Tradeoff: token now visible in 'ps aux' for the duration of the git child process (same as before via env var), but no longer in any persistent state. (2) CACHE-VALIDITY FOOTGUN — fixed Before: cache-hit was 'cacheDir/.git exists'. A clone interrupted after .git was created but before content finished writing would leave a partially-written cache that subsequent imports treated as hit, returning stale/incomplete content forever (no self-heal). After: cache-hit also requires a .complete marker file written only AFTER successful clone+rename. Partially-written cache is treated as cache-miss and re-fetched cleanly (after RemoveAll on the partial dir to avoid blocking the new clone's mkdir). (3) REF '..' DENY — fixed Before: safeRefPattern '^[a-zA-Z0-9_./-]+$' allowed '..' as a substring. Git itself rejects most refs containing '..', but defense-in-depth says don't depend on the downstream tool's validation when sanitizing input at the boundary. After: explicit strings.Contains(ref.Ref, '..') check. (4) NAMING CLEANUP — fixed Before: rewriteFilesDirAndIncludes() — name claims to rewrite !include scalars but doesn't (we removed that during PR-A development; double-prefix bug). Misleading for readers. After: rewriteFilesDir(). Docstring updated to explicitly explain why !include paths are NOT rewritten (relative to subDir, naturally inside cache). Also: removed unused buildAuthedURL() (replaced by buildExternalCloneURL + authConfigArgs split), removed unused shortHash() helper (replaced by os.MkdirTemp), removed unused crypto/sha1 + encoding/hex + fmt imports, removed stray '_ = fmt.Sprint' line in integration test. NEW TESTS - TestGitFetcher_RejectsRefWithDoubleDot (defense-in-depth on ref input) - TestGitFetcher_CacheValidatedByCompleteMarker (partial cache → re-fetch) VERIFIED LOCALLY 2026-05-08 Full ./internal/handlers/ suite: ok (7.8s, 14 external-resolver tests + all existing tests). Two new tests cover the two new behaviors. Refs: internal#77 — extraction RFC molecule-core#105 (resolver), #106 (tests) — original implementation Hongming code-review-and-quality skill invocation 2026-05-08 + 'fix all'	2026-05-08 05:54:54 -07:00
claude-ceo-assistant	d9056db5b4	Merge pull request 'test(org-external): integration + e2e for !external resolver (PR-B + PR-C)' (#106) from feature/external-ref-pr-bc-tests into staging	2026-05-08 12:33:45 +00:00
claude-ceo-assistant	89c5567d79	test(org-external): integration test against local bare-git + e2e against live Gitea (PR-B + PR-C) ... PR-B (local bare-git integration, task #233): workspace-server/internal/handlers/org_external_integration_test.go Three tests using git's GIT_CONFIG_COUNT/KEY/VALUE env-var-injected insteadOf URL rewrite — process-scoped, no ~/.gitconfig pollution: - TestGitFetcher_RealClone_LocalRedirect: full resolver chain end-to- end with REAL git clone against a local bare-repo, asserts cache population + content materialization + path rewrite + cache-hit on second invocation. - TestGitFetcher_RealClone_BadRefFails: nonexistent ref surfaces git's error cleanly through the ls-remote step. - TestGitFetcher_DirectFetch_CacheHit: gitFetcher.Fetch direct invocation (no resolver wrapping); verifies cache-hit returns same dir + same SHA, no clobber. Production code untouched — insteadOf rewrite makes the production gitFetcher think it's cloning from Gitea, but git rewrites at clone time to file://<barePath>. Tests the real shell-out + parsing. PR-C (live Gitea e2e, task #234): workspace-server/internal/handlers/local_e2e_dev_dept_test.go TestLocalE2E_ExternalDevDepartment — minimal parent template that uses !external against the LIVE molecule-ai/molecule-dev-department repo. No symlink, no /tmp/local-e2e-deploy fixture. Composition resolves over network at import time. Asserts: - 28+ dev-tree workspaces resolve through the fetched cache (matches the count from TestLocalE2E_DevDepartmentExtraction) - Q1 placement: 'Documentation Specialist' present (under app-lead) - Q2 placement: 'Triage Operator' present (under dev-lead) - Every workspace's files_dir is cache-prefixed (proves rewrite ran) - Every workspace's resolveInsideRoot+Stat succeeds (would fail provisioning if not) Skipped if Gitea unreachable (TCP probe to git.moleculesai.app:443) or git binary absent — won't false-fail offline runners. VERIFIED LOCALLY 2026-05-08: --- PASS: TestGitFetcher_RealClone_LocalRedirect (0.26s) --- PASS: TestGitFetcher_RealClone_BadRefFails (0.15s) --- PASS: TestGitFetcher_DirectFetch_CacheHit (0.23s) --- PASS: TestLocalE2E_ExternalDevDepartment (0.55s) workspaces resolved through !external: 28 Full ./internal/handlers/ test suite: ok (no regressions) Together with PR-A's unit tests (#105), the !external resolver is now covered at three layers: - unit (fakeFetcher injection): allowlist, validation, path rewrite - integration (real git, local bare-repo): clone, cache, ls-remote - e2e (real git, live Gitea, live dev-department): full chain Refs: internal#77 — extraction RFC (Phase 3a phasing in comment 1995) task #233 (PR-B), task #234 (PR-C) Hongming GO 2026-05-08 ('do PR-B/C/D')	2026-05-08 05:30:04 -07:00
claude-ceo-assistant	ef0ef30116	Merge pull request 'feat(org-import): !external cross-repo subtree resolver (Phase 3a, task #222)' (#105) from feature/external-ref-resolver into staging	2026-05-08 12:23:33 +00:00
claude-ceo-assistant	257d6c1b5a	feat(org-import): !external cross-repo subtree resolver (Phase 3a, internal#77 / task #222) ... Adds gitops-style cross-repo subtree composition to the platform's org-template importer. Replaces (eventually) the operator-side filesystem symlink approach shipped in PR #5. DESIGN See internal#77 comment 1995 for the full design doc + decision points agreed with Hongming 2026-05-08. Schema: a `!external`-tagged mapping anywhere a workspace entry is allowed (workspaces:, roots:, children:): - !external repo: molecule-ai/molecule-dev-department ref: main path: dev-lead/workspace.yaml url: git.moleculesai.app # optional; default = MOLECULE_EXTERNAL_GITEA_URL or git.moleculesai.app At resolve time the platform fetches the repo at ref into a content- addressable cache under <orgBaseDir>/.external-cache/<repo>/<sha>/, loads <cacheDir>/<path>, recursively resolves nested !include / !external in the loaded subtree, then rewrites every files_dir scalar in the fully-resolved subtree to be cache-prefixed. Downstream pipeline (resolveInsideRoot, plugin merge, CopyTemplateToContainer) sees ordinary in-tree paths. IMPLEMENTATION - org_external.go: ExternalRef type, fetcher interface (gitFetcher production + injectable for tests), resolveExternalMapping resolver, rewriteFilesDirAndIncludes path-rewrite walker, allowlistedHostPath + safeRefPattern + safeRepoCacheDir validation helpers. - org_include.go: 4-line hook in expandNode dispatching MappingNode with Tag=="!external" to resolveExternalMapping. - org_external_test.go: 8 unit tests with fakeFetcher injection (no network): * happy path (top + nested workspace files_dir cache-prefixed) * allowlist rejection (github.com/foo/bar) * path-traversal rejection (../../etc/passwd) * malformed ref rejection ("main; rm -rf /") * missing required fields (repo / ref / path) * rewriteFilesDirAndIncludes basic + idempotent * allowlistedHostPath env-override + glob Path rewrite ONLY rewrites files_dir scalars. !include scalars are NOT rewritten — they resolve relative to their containing file's directory, which post-fetch is naturally inside the cache, so relative !includes Just Work without modification. ALLOWLIST + AUTH - Default allowlist: git.moleculesai.app/molecule-ai/. - Override: MOLECULE_EXTERNAL_REPO_ALLOWLIST (comma-separated prefixes; trailing /* or / supported). - Auth: MOLECULE_GITEA_TOKEN env var injected into clone URL. Optional — falls back to unauthenticated for public repos. - Reject: malformed refs, path-traversal, non-allowlisted hosts. CACHE - Location: <orgBaseDir>/.external-cache/<safe-repo>/<sha>/. Operators add to .gitignore. - Content-addressable: same (repo, sha) reuses cache, no overwrite. - Atomic clone via tmp-then-rename. - Concurrency: race-tolerant — last-writer-wins on same SHA. GC out of scope for v1 (filed as parked follow-up). SECURITY (per SOP Phase 2) Untrusted yaml input — all validated: repo: allowlist (default molecule-ai/* on Gitea host) ref: ^[a-zA-Z0-9_./-]+$ regex (rejects shell injection) path: relative-and-down-only (rejects ../escape) Auth: read-only token scoped to allowed orgs. Recursion: maxExternalDepth=4 (vs maxIncludeDepth=16) to limit network fan-out cost. Cache poisoning: per-(repo, sha) content-addressable; can't poison across SHAs. Trust boundary: cloned content treated identically to a sibling- cloned subtree (same model as current symlink approach). VERSIONING / BACKWARDS COMPAT Pure additive. Existing !include and inline workspaces unchanged. Existing dev-lead symlink (parent template PR #5) keeps working. Migration of parent template to !external is a separate PR-D. No DB schema change. No public API change. VERIFIED LOCALLY go test ./internal/handlers/ → ok (5.2s, all 8 new tests + existing) Stub fetcher injection lets unit tests cover the resolver + path-rewrite logic without network. PR-B (follow-up) adds an integration test against a local bare-git repo. PR-C adds the real-Gitea e2e test against the live dev-department repo. Refs: internal#77 — extraction RFC (comment 1995 = Phase 1+2 design) task #222 — this PR is Phase 3a (PR-A in the design's phasing) Hongming GO 2026-05-08 ('go' on 4 decision points + design)	2026-05-08 05:17:55 -07:00
claude-ceo-assistant	7b6061e899	Merge pull request 'fix(provisioner)+test: EvalSymlinks templatePath; stage-2 e2e for files_dir consumption' (#104) from optimize/extraction-stage-2-tests-and-validator-fix into staging	2026-05-08 11:49:35 +00:00
claude-ceo-assistant	3dcc7230f9	fix(provisioner)+test: EvalSymlinks templatePath; stage-2 e2e for files_dir consumption ... Two changes that fall out of one root cause discovered while preparing the local platform spin-up for the dev-department extraction (internal#77): PROBLEM CopyTemplateToContainer's filepath.Walk is called with templatePath set to the workspace's resolved files_dir. With the cross-repo symlink composition shipped in PR #5 (parent template's dev-lead → ../molecule-dev-department/dev-lead/), the Dev Lead workspace's files_dir is literally 'dev-lead' — i.e. the symlink itself, not a path THROUGH the symlink. filepath.Walk does not descend into a symlink leaf — it Lstats the root, sees a symlink (mode bit set, not a directory), emits exactly one entry, and returns. Result: the workspace's /configs/ tar would ship empty. Other 38 workspaces are fine because their files_dir paths just TRAVERSE the symlink (path resolution handles intermediate symlinks via Lstat traversal); only the leaf-is-symlink case breaks. FIX workspace-server/internal/provisioner/provisioner.go: Call filepath.EvalSymlinks on templatePath before filepath.Walk. Resolves the leaf-symlink case for ALL templates, not just dev-dept. Security: templatePath has already passed resolveInsideRoot's path-string check at the call site; the trust boundary is the operator-side /org-templates/ filesystem layout, not this resolution step. TEST workspace-server/internal/handlers/local_e2e_dev_dept_test.go: New TestLocalE2E_FilesDirConsumption — stage-2 of the local e2e. For every workspace in the resolved OrgTemplate, asserts: 1. resolveInsideRoot(orgBaseDir, ws.FilesDir) succeeds. 2. os.Stat on the result returns a directory. 3. filepath.Walk after EvalSymlinks (mirroring the platform fix) emits at least one file. 4. At least one workspace marker exists (workspace.yaml, system-prompt.md, or initial-prompt.md). Exercises the SECOND half of POST /org/import that TestLocalE2E_DevDepartmentExtraction (PR #103) didn't cover. VERIFIED LOCALLY (2026-05-08, against post-extraction Gitea state): --- PASS: TestLocalE2E_FilesDirConsumption (0.05s) checked 39 workspaces with files_dir All 39 walk paths emit non-empty file sets with valid workspace markers. REGRESSION GUARD Without the EvalSymlinks fix, this test fails on Dev Lead with: files_dir 'dev-lead' at '/.../molecule-dev/dev-lead' is empty — CopyTemplateToContainer would produce empty /configs/ Refs: internal#77 — extraction RFC molecule-core#102 (resolver symlink contract test) molecule-core#103 (stage-1 e2e: include resolution) Hongming GO 2026-05-08 ('go' on the 3 pre-spin-up optimizations)	2026-05-08 04:46:33 -07:00
claude-ceo-assistant	e3f17fb954	Merge pull request 'test(local-e2e): verify dev-department extraction end-to-end via real resolveYAMLIncludes' (#103) from test/e2e-dev-department-extraction into staging	2026-05-08 11:29:33 +00:00
claude-ceo-assistant	3adbbacf2e	test(local-e2e): verify dev-department extraction end-to-end via real resolveYAMLIncludes ... Phase 4 (local-only) of internal#77 (dev-department extraction). Adds TestLocalE2E_DevDepartmentExtraction that exercises the FULL platform import path against the real molecule-ai-org-template-molecule-dev (post-slim) and molecule-ai/molecule-dev-department (post-atomize) repos cloned as siblings under /tmp/local-e2e-deploy/. What it proves end-to-end: - The dev-lead symlink at parent's template root is followed by resolveYAMLIncludes (filepath.Abs/Rel-style security check passes, os.ReadFile follows the link). - Recursive !include chain through the symlinked subtree resolves: parent's org.yaml → !include dev-lead/workspace.yaml (symlinked) → !include ./core-lead/workspace.yaml → !include ./core-be/workspace.yaml (atomized children: paths, no '..'). - 39 workspaces enumerate after resolution: 5 PM-tree + 6 Marketing-tree + 28 dev-tree (Dev Lead + 5 sub-team leads + 18 leaf workspaces + 3 floaters + 1 triage-operator). - Q1+Q2 placements verified by sentinel name check: 'Documentation Specialist' is reachable (under app-lead via app-docs sub-team), 'Triage Operator' is reachable (direct child of Dev Lead). Test skips with t.Skipf if the local-e2e fixture isn't present on the host — won't block CI on hosts that haven't set it up. To set up locally: TESTROOT=/tmp/local-e2e-deploy mkdir -p $TESTROOT && cd $TESTROOT git clone https://git.moleculesai.app/molecule-ai/molecule-ai-org-template-molecule-dev.git molecule-dev git clone https://git.moleculesai.app/molecule-ai/molecule-dev-department.git cd /Users/<you>/molecule-core/workspace-server go test -v -run TestLocalE2E_DevDepartmentExtraction ./internal/handlers/ Verified locally 2026-05-08: --- PASS: TestLocalE2E_DevDepartmentExtraction (0.01s) total workspaces (recursive): 39 Refs: internal#77 — extraction RFC molecule-core PR #102 — symlink-resolution contract test molecule-ai/molecule-dev-department PRs #1, #2, #3 (scaffold + extract + atomize) molecule-ai/molecule-ai-org-template-molecule-dev PR #5 (parent slim + symlink wire) Hongming GO 2026-05-08 ('lets not go for staging right now, we do local test first') SOP Phase 4 (local) — task #226	2026-05-08 04:24:47 -07:00
claude-ceo-assistant	1bd80defab	Merge pull request 'test(org-include): pin symlink-based subtree composition contract' (#102) from fix/org-import-subtree-symlink-test into staging	2026-05-08 11:23:05 +00:00
claude-ceo-assistant	78c4b9b74f	test(org-include): pin symlink-based subtree composition contract ... Two new tests in workspace-server/internal/handlers/org_include_test.go: - TestResolveYAMLIncludes_FollowsDirectorySymlink: parent template's org.yaml `!include`s into a sibling-repo subtree via a relative directory symlink. The resolver's filepath.Abs/Rel security check operates on path strings (passes), and os.ReadFile follows the symlink at OS layer (file content delivered). Recursive nested `!include`s within the symlinked subtree resolve correctly because filepath.Dir(absTarget) keeps the literal symlink path as currentDir. - TestResolveYAMLIncludes_RejectsSymlinkEscapingRoot: companion test pinning current behavior where a symlink target outside the parent root is followed (resolveInsideRoot doesn't EvalSymlinks). Asserted as 'should resolve' so future hardening (if filepath.EvalSymlinks is added) flips the test red and forces a coordinated update to the dev-department subtree-composition pattern. Why now: internal#77 RFC (dev-department extraction) selects symlink- based composition over a future platform-level external: ref. These tests pin the contract before the operator-side symlink convention gets shipped, so a refactor or hardening of the resolver can't silently break the production org-import path. No production code changes. Pure additive test coverage. Refs: internal#77 (Phase 3b verification — task #223)	2026-05-07 20:42:38 -07:00