From 50990c79f7f4eac9a89ed7d0218a60c73b60cd7d Mon Sep 17 00:00:00 2001 From: Hongming Wang Date: Thu, 16 Apr 2026 00:32:09 -0700 Subject: [PATCH] =?UTF-8?q?feat(org-templates):=20Phase=201=20=E2=80=94=20?= =?UTF-8?q?externalize=20prompt=20bodies=20to=20sibling=20files=20(#389)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part 1 of 4 in the scalability refactor. Each role can now keep its initial_prompt / idle_prompt / schedule prompts as sibling .md files under files_dir/; inline YAML literals still work for backwards-compat. ## What changes **Platform (org.go importer):** - `OrgWorkspace` gains `InitialPromptFile`, `IdlePrompt`, `IdlePromptFile`, `IdleIntervalSeconds`. The idle_* fields were previously dropped by the org importer entirely — struct didn't declare them — which is why engineer idle_prompts never propagated from org.yaml to live /configs (I've been manually docker-cp'ing them in every maintenance cron). - `OrgSchedule` gains `PromptFile`. Hourly/weekly cron prompts are the largest bodies in org.yaml (1-5 KB each) and get resolved at import time just like initial_prompt. - `OrgDefaults` gains the same idle_* + *_file fields for org-wide fallback. - New `resolvePromptRef(inline, fileRef, orgBaseDir, filesDir)` helper — the single chokepoint for inline-vs-file resolution. Inline wins when both are set. File refs route through `resolveInsideRoot` so a crafted ref can't escape the org template directory (same traversal defense as files_dir). - `createWorkspaceTree` now injects idle_prompt + idle_interval_seconds into the workspace's config.yaml (previously missing — that's the second half of the idle-prompt propagation bug). **Tests:** - `org_prompt_ref_test.go` — 10 cases: inline-wins, file-read-when-empty, both-empty, defaults-level resolution, inline-template mode errors, traversal rejection (via file ref AND via files_dir), missing-file errors, and YAML-unmarshal parsing for each new field. **Proof migration:** - Documentation Specialist (biggest role at 6.9 KB of prompts) moves from inline YAML to `documentation-specialist/{initial-prompt.md, schedules/daily-docs-sync.md, schedules/weekly-terminology-audit.md}`. - org.yaml drops 1801 → 1687 lines (-6.3%) from just this one role. ## Why this matters org.yaml is 108 KB of which 67 KB (62%) is prompt text. At the current 12-role template size that's already unreadable; the marketing + triage- operator additions pushed it to 1801 lines. The 4-phase refactor aims: - **Phase 1 (this PR):** platform support + 1 role proof. - **Phase 2:** migrate remaining ~20 roles to file refs. Target: org.yaml at ~600 lines of pure structural scaffolding. - **Phase 3:** YAML `!include` preprocessor — split org.yaml into teams/{research,dev,marketing,ops}.yaml shards. - **Phase 4:** per-workspace atomization — each role gets its own workspace.yaml manifest; org.yaml composes them. ## Backwards compatibility - Inline `initial_prompt: |` / `prompt: |` / `idle_prompt: |` all still work. - Missing `prompt_file` refs log + skip the schedule (not fatal) — fail loud so bugs surface during deployment rather than silent-drop. - Inline-template mode (POST /org/import with raw JSON body, no `dir`) errors cleanly when a file ref is used — can't resolve files without a base dir, surface that rather than guessing. ## Test plan - [x] `go build ./...` clean - [x] `go test -run 'TestResolvePromptRef|TestOrgYAML' ./internal/handlers/` — 10 tests pass - [x] `python -c "yaml.safe_load(...)"` on the edited org.yaml — parses - [ ] Post-merge: deploy platform rebuild, run `POST /org/import` against a fresh workspace, verify Documentation Specialist's /configs/config.yaml contains the initial_prompt body and workspace_schedules rows contain the cron prompts (phantom-success check: grep the actual content, not just the row count). Co-authored-by: Claude Opus 4.6 (1M context) --- .../initial-prompt.md | 36 +++ .../schedules/daily-docs-sync.md | 74 ++++++ .../schedules/weekly-terminology-audit.md | 28 +++ org-templates/molecule-dev/org.yaml | 150 +----------- platform/internal/handlers/org.go | 211 ++++++++++++++--- .../internal/handlers/org_prompt_ref_test.go | 221 ++++++++++++++++++ 6 files changed, 549 insertions(+), 171 deletions(-) create mode 100644 org-templates/molecule-dev/documentation-specialist/initial-prompt.md create mode 100644 org-templates/molecule-dev/documentation-specialist/schedules/daily-docs-sync.md create mode 100644 org-templates/molecule-dev/documentation-specialist/schedules/weekly-terminology-audit.md create mode 100644 platform/internal/handlers/org_prompt_ref_test.go diff --git a/org-templates/molecule-dev/documentation-specialist/initial-prompt.md b/org-templates/molecule-dev/documentation-specialist/initial-prompt.md new file mode 100644 index 00000000..ecec7e6d --- /dev/null +++ b/org-templates/molecule-dev/documentation-specialist/initial-prompt.md @@ -0,0 +1,36 @@ +You just started as Documentation Specialist. Set up silently — do NOT contact other agents. + +⚠️ PRIVACY RULE (read first, never violate): +molecule-controlplane is a PRIVATE repo. Its source code, file paths, +internal endpoints, schema details, infra config, billing/auth +implementation — none of that goes into the public docs site +(Molecule-AI/docs) or the public README in molecule-monorepo. Public +docs may describe the SaaS PRODUCT (signup, billing, tenant isolation +guarantees) but never the provisioner's internals. When in doubt: +don't publish. + +1. Clone all three repos: + git clone https://github.com/${GITHUB_REPO}.git /workspace/repo 2>/dev/null || (cd /workspace/repo && git pull) + git clone https://github.com/Molecule-AI/docs.git /workspace/docs 2>/dev/null || (cd /workspace/docs && git pull) + git clone https://github.com/Molecule-AI/molecule-controlplane.git /workspace/controlplane 2>/dev/null || (cd /workspace/controlplane && git pull) +2. Read /workspace/repo/CLAUDE.md — full architecture, what's public-facing +3. Read /configs/system-prompt.md +4. Read /workspace/docs/README.md and /workspace/docs/content/docs/index.mdx +5. Read /workspace/controlplane/README.md and /workspace/controlplane/PLAN.md + — understand what the SaaS provisioner does (private) vs what users see (public) +6. Run: cd /workspace/docs && ls content/docs/*.mdx + — note which pages are stubs ("Coming soon" marker) vs hand-written +7. Run: cd /workspace/repo && git log --oneline -20 -- platform/internal/handlers/ org-templates/ plugins/ + — note recent public-surface changes in the platform repo +8. Run: cd /workspace/controlplane && git log --oneline -20 + — note recent controlplane changes (these need internal docs only) +9. Use commit_memory to save: + - Stubs that need backfilling (docs site) + - Recent platform PRs that have NO docs PR yet + - Recent controlplane PRs whose internal README needs an update + - Public concepts that lack a canonical naming entry +10. Wait for tasks from PM. Your owned surfaces are: + - https://github.com/Molecule-AI/docs (customer site, Fumadocs) — PUBLIC + - /workspace/repo/docs/ (internal architecture / edit-history) — PUBLIC + - /workspace/repo/README.md and per-package READMEs — PUBLIC + - /workspace/controlplane/README.md, PLAN.md, internal docs — PRIVATE diff --git a/org-templates/molecule-dev/documentation-specialist/schedules/daily-docs-sync.md b/org-templates/molecule-dev/documentation-specialist/schedules/daily-docs-sync.md new file mode 100644 index 00000000..1d2533c3 --- /dev/null +++ b/org-templates/molecule-dev/documentation-specialist/schedules/daily-docs-sync.md @@ -0,0 +1,74 @@ +Daily documentation maintenance. Two parallel objectives: +(1) keep the public docs site current with the platform repo, +(2) backfill stub pages on the docs site one at a time. + +SETUP: + cd /workspace/repo && git pull 2>/dev/null || true + cd /workspace/docs && git pull 2>/dev/null || true + cd /workspace/controlplane && git pull 2>/dev/null || true + +1a. PAIR RECENT PLATFORM PRS (last 24h): + cd /workspace/repo + gh pr list --repo Molecule-AI/molecule-monorepo --state merged \ + --search "merged:>$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \ + --json number,title,files + For each merged PR that touches a public surface + (platform/internal/handlers/, plugins/*, org-templates/*, + docs/architecture.md, README.md, workspace-template/adapters/*): + - Identify which docs page(s) on the public site cover that surface. + - If a docs page exists but is stale → update it with examples + from the PR diff. Open a PR to Molecule-AI/docs with the change. + - If NO docs page exists for the new surface → propose one + (add to content/docs/meta.json + new .mdx file). Open a PR. + - Always close PRs with `Closes platform PR #N` so the link is durable. + +1b. PAIR RECENT CONTROLPLANE PRS (last 24h): + cd /workspace/controlplane + gh pr list --repo Molecule-AI/molecule-controlplane --state merged \ + --search "merged:>$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \ + --json number,title,files + ⚠️ PRIVATE REPO. Two cases: + (i) Internal-only change (handler, schema, infra, fly.toml, + billing logic): update README.md + PLAN.md + any + docs/internal/*.md inside molecule-controlplane itself. + Open the PR against Molecule-AI/molecule-controlplane. + NEVER mention these changes in /workspace/docs. + (ii) Customer-facing change (new tier, new region, new SLA, + pricing change, signup flow change): write a sanitized + description for the PUBLIC docs site (e.g. "We now offer + EU-region tenants" — NOT "controlplane reads FLY_REGION + from env and passes it to provisioner.go:142"). Open a + PR against Molecule-AI/docs. + When unsure which category a change falls into: default to + INTERNAL-only and ask PM for explicit approval before publishing. + +2. BACKFILL ONE STUB PAGE: + cd /workspace/docs + grep -l "Coming soon" content/docs/*.mdx | head -1 + Pick the highest-priority stub (one of: org-template, plugins, + channels, schedules, architecture, api-reference, self-hosting, + observability, troubleshooting). Write 300-800 words of + hand-crafted, example-rich content based on: + - The actual code in /workspace/repo/platform/internal/handlers/ + - The actual templates in /workspace/repo/org-templates/ + - The actual plugin manifests in /workspace/repo/plugins/ + Cite file paths so readers can follow the source. Open a PR. + +3. LINK + ANCHOR CHECK: + Use the browser-automation plugin to crawl + https://doc.moleculesai.app (or the local dev server if the + site isn't deployed yet — `cd /workspace/docs && npm install + && npm run build && npm run start`). Report broken links and + missing anchors back to PM. + +4. ROUTING: + delegate_task to PM with audit_summary metadata: + - category: docs + - severity: info + - issues: [list of PR numbers opened to Molecule-AI/docs] + - top_recommendation: one-line summary + If nothing to do today, PM-message a one-line "clean". + +5. MEMORY: + Save key 'docs-sync-latest' with timestamp + list of stub + pages still pending + count of paired PRs this cycle. diff --git a/org-templates/molecule-dev/documentation-specialist/schedules/weekly-terminology-audit.md b/org-templates/molecule-dev/documentation-specialist/schedules/weekly-terminology-audit.md new file mode 100644 index 00000000..9178c986 --- /dev/null +++ b/org-templates/molecule-dev/documentation-specialist/schedules/weekly-terminology-audit.md @@ -0,0 +1,28 @@ +Weekly audit of documentation freshness and terminology consistency. + +1. STALE PAGE DETECTION: + cd /workspace/docs && for f in content/docs/*.mdx; do + age=$(git log -1 --format='%cr' -- "$f") + echo "$age :: $f" + done | sort -r + Flag any page not touched in 30+ days that covers a + fast-moving surface (handlers, plugins, templates). + +2. TERMINOLOGY CONSISTENCY: + grep -rEi "workspace|agent|cron|schedule|plugin|channel|template" \ + content/docs/*.mdx | grep -oE "\b(workspace|workspaces|Agent|agent|cron job|schedule|plugin|channel|template)\b" | \ + sort | uniq -c | sort -rn + Each concept should have ONE canonical capitalisation and + plural form. Open a PR fixing inconsistencies. + +3. LINK ROT: + grep -rE "\[.*\]\(http[^)]+\)" content/docs/*.mdx | \ + awk -F'[()]' '{print $2}' | sort -u | \ + while read url; do + curl -sIo /dev/null -w "%{http_code} $url\n" "$url" + done | grep -v "^200 " + Report any non-200 to PM. + +4. ROUTING + MEMORY: + Same audit_summary contract as the daily cron. + Save findings to memory key 'docs-weekly-audit'. diff --git a/org-templates/molecule-dev/org.yaml b/org-templates/molecule-dev/org.yaml index a81cff18..ae601dbc 100644 --- a/org-templates/molecule-dev/org.yaml +++ b/org-templates/molecule-dev/org.yaml @@ -1150,153 +1150,21 @@ workspaces: # docs site (visual regressions, broken links, dead anchors) plus # update-docs skill (already in defaults) for cross-repo docs sync. plugins: [browser-automation] - initial_prompt: | - You just started as Documentation Specialist. Set up silently — do NOT contact other agents. - - ⚠️ PRIVACY RULE (read first, never violate): - molecule-controlplane is a PRIVATE repo. Its source code, file paths, - internal endpoints, schema details, infra config, billing/auth - implementation — none of that goes into the public docs site - (Molecule-AI/docs) or the public README in molecule-monorepo. Public - docs may describe the SaaS PRODUCT (signup, billing, tenant isolation - guarantees) but never the provisioner's internals. When in doubt: - don't publish. - - 1. Clone all three repos: - git clone https://github.com/${GITHUB_REPO}.git /workspace/repo 2>/dev/null || (cd /workspace/repo && git pull) - git clone https://github.com/Molecule-AI/docs.git /workspace/docs 2>/dev/null || (cd /workspace/docs && git pull) - git clone https://github.com/Molecule-AI/molecule-controlplane.git /workspace/controlplane 2>/dev/null || (cd /workspace/controlplane && git pull) - 2. Read /workspace/repo/CLAUDE.md — full architecture, what's public-facing - 3. Read /configs/system-prompt.md - 4. Read /workspace/docs/README.md and /workspace/docs/content/docs/index.mdx - 5. Read /workspace/controlplane/README.md and /workspace/controlplane/PLAN.md - — understand what the SaaS provisioner does (private) vs what users see (public) - 6. Run: cd /workspace/docs && ls content/docs/*.mdx - — note which pages are stubs ("Coming soon" marker) vs hand-written - 7. Run: cd /workspace/repo && git log --oneline -20 -- platform/internal/handlers/ org-templates/ plugins/ - — note recent public-surface changes in the platform repo - 8. Run: cd /workspace/controlplane && git log --oneline -20 - — note recent controlplane changes (these need internal docs only) - 9. Use commit_memory to save: - - Stubs that need backfilling (docs site) - - Recent platform PRs that have NO docs PR yet - - Recent controlplane PRs whose internal README needs an update - - Public concepts that lack a canonical naming entry - 10. Wait for tasks from PM. Your owned surfaces are: - - https://github.com/Molecule-AI/docs (customer site, Fumadocs) — PUBLIC - - /workspace/repo/docs/ (internal architecture / edit-history) — PUBLIC - - /workspace/repo/README.md and per-package READMEs — PUBLIC - - /workspace/controlplane/README.md, PLAN.md, internal docs — PRIVATE + # Phase 1 scalability: prompts externalized to sibling .md files. + # See documentation-specialist/{initial-prompt.md, schedules/*.md}. + # The platform's org importer reads these at POST /org/import time + # and inlines them into the workspace's /configs/config.yaml and + # workspace_schedules rows. Inline `initial_prompt:` / `prompt:` + # still win if both are set (backwards-compat). + initial_prompt_file: initial-prompt.md schedules: - name: Daily docs sync — backfill stubs and pair recent platform PRs cron_expr: "0 9 * * *" - prompt: | - Daily documentation maintenance. Two parallel objectives: - (1) keep the public docs site current with the platform repo, - (2) backfill stub pages on the docs site one at a time. - - SETUP: - cd /workspace/repo && git pull 2>/dev/null || true - cd /workspace/docs && git pull 2>/dev/null || true - cd /workspace/controlplane && git pull 2>/dev/null || true - - 1a. PAIR RECENT PLATFORM PRS (last 24h): - cd /workspace/repo - gh pr list --repo Molecule-AI/molecule-monorepo --state merged \ - --search "merged:>$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \ - --json number,title,files - For each merged PR that touches a public surface - (platform/internal/handlers/, plugins/*, org-templates/*, - docs/architecture.md, README.md, workspace-template/adapters/*): - - Identify which docs page(s) on the public site cover that surface. - - If a docs page exists but is stale → update it with examples - from the PR diff. Open a PR to Molecule-AI/docs with the change. - - If NO docs page exists for the new surface → propose one - (add to content/docs/meta.json + new .mdx file). Open a PR. - - Always close PRs with `Closes platform PR #N` so the link is durable. - - 1b. PAIR RECENT CONTROLPLANE PRS (last 24h): - cd /workspace/controlplane - gh pr list --repo Molecule-AI/molecule-controlplane --state merged \ - --search "merged:>$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \ - --json number,title,files - ⚠️ PRIVATE REPO. Two cases: - (i) Internal-only change (handler, schema, infra, fly.toml, - billing logic): update README.md + PLAN.md + any - docs/internal/*.md inside molecule-controlplane itself. - Open the PR against Molecule-AI/molecule-controlplane. - NEVER mention these changes in /workspace/docs. - (ii) Customer-facing change (new tier, new region, new SLA, - pricing change, signup flow change): write a sanitized - description for the PUBLIC docs site (e.g. "We now offer - EU-region tenants" — NOT "controlplane reads FLY_REGION - from env and passes it to provisioner.go:142"). Open a - PR against Molecule-AI/docs. - When unsure which category a change falls into: default to - INTERNAL-only and ask PM for explicit approval before publishing. - - 2. BACKFILL ONE STUB PAGE: - cd /workspace/docs - grep -l "Coming soon" content/docs/*.mdx | head -1 - Pick the highest-priority stub (one of: org-template, plugins, - channels, schedules, architecture, api-reference, self-hosting, - observability, troubleshooting). Write 300-800 words of - hand-crafted, example-rich content based on: - - The actual code in /workspace/repo/platform/internal/handlers/ - - The actual templates in /workspace/repo/org-templates/ - - The actual plugin manifests in /workspace/repo/plugins/ - Cite file paths so readers can follow the source. Open a PR. - - 3. LINK + ANCHOR CHECK: - Use the browser-automation plugin to crawl - https://doc.moleculesai.app (or the local dev server if the - site isn't deployed yet — `cd /workspace/docs && npm install - && npm run build && npm run start`). Report broken links and - missing anchors back to PM. - - 4. ROUTING: - delegate_task to PM with audit_summary metadata: - - category: docs - - severity: info - - issues: [list of PR numbers opened to Molecule-AI/docs] - - top_recommendation: one-line summary - If nothing to do today, PM-message a one-line "clean". - - 5. MEMORY: - Save key 'docs-sync-latest' with timestamp + list of stub - pages still pending + count of paired PRs this cycle. + prompt_file: schedules/daily-docs-sync.md enabled: true - name: Weekly terminology + freshness audit cron_expr: "0 11 * * 1" - prompt: | - Weekly audit of documentation freshness and terminology consistency. - - 1. STALE PAGE DETECTION: - cd /workspace/docs && for f in content/docs/*.mdx; do - age=$(git log -1 --format='%cr' -- "$f") - echo "$age :: $f" - done | sort -r - Flag any page not touched in 30+ days that covers a - fast-moving surface (handlers, plugins, templates). - - 2. TERMINOLOGY CONSISTENCY: - grep -rEi "workspace|agent|cron|schedule|plugin|channel|template" \ - content/docs/*.mdx | grep -oE "\b(workspace|workspaces|Agent|agent|cron job|schedule|plugin|channel|template)\b" | \ - sort | uniq -c | sort -rn - Each concept should have ONE canonical capitalisation and - plural form. Open a PR fixing inconsistencies. - - 3. LINK ROT: - grep -rE "\\[.*\\]\\(http[^)]+\\)" content/docs/*.mdx | \ - awk -F'[()]' '{print $2}' | sort -u | \ - while read url; do - curl -sIo /dev/null -w "%{http_code} $url\n" "$url" - done | grep -v "^200 " - Report any non-200 to PM. - - 4. ROUTING + MEMORY: - Same audit_summary contract as the daily cron. - Save findings to memory key 'docs-weekly-audit'. + prompt_file: schedules/weekly-terminology-audit.md enabled: true - name: Triage Operator diff --git a/platform/internal/handlers/org.go b/platform/internal/handlers/org.go index f00a4ca9..f5786b2e 100644 --- a/platform/internal/handlers/org.go +++ b/platform/internal/handlers/org.go @@ -81,11 +81,26 @@ type OrgTemplate struct { } type OrgDefaults struct { - Runtime string `yaml:"runtime" json:"runtime"` - Tier int `yaml:"tier" json:"tier"` - Model string `yaml:"model" json:"model"` - Plugins []string `yaml:"plugins" json:"plugins"` - InitialPrompt string `yaml:"initial_prompt" json:"initial_prompt"` + Runtime string `yaml:"runtime" json:"runtime"` + Tier int `yaml:"tier" json:"tier"` + Model string `yaml:"model" json:"model"` + Plugins []string `yaml:"plugins" json:"plugins"` + InitialPrompt string `yaml:"initial_prompt" json:"initial_prompt"` + // InitialPromptFile is a file ref alternative to InitialPrompt. Path is + // resolved relative to the workspace's files_dir (or the org base dir + // when used at defaults level — defaults don't have their own files_dir, + // so the file must live at the org root). Inline InitialPrompt wins + // when both are set. + InitialPromptFile string `yaml:"initial_prompt_file" json:"initial_prompt_file"` + // IdlePrompt / IdleIntervalSeconds are the workspace-default idle-loop + // body and cadence (see workspace-template/heartbeat.py). They were + // previously dropped by the org importer because the struct didn't + // declare them — causing live configs to boot without idle_prompts + // even when org.yaml had them. Phase 1 scalability work adds both + // inline + file-ref forms. + IdlePrompt string `yaml:"idle_prompt" json:"idle_prompt"` + IdlePromptFile string `yaml:"idle_prompt_file" json:"idle_prompt_file"` + IdleIntervalSeconds int `yaml:"idle_interval_seconds" json:"idle_interval_seconds"` // CategoryRouting maps issue/audit category → list of target roles. // Per-workspace blocks UNION + override per-key with these defaults. // Rendered into each workspace's config.yaml so agent prompts can read it @@ -98,7 +113,13 @@ type OrgSchedule struct { CronExpr string `yaml:"cron_expr" json:"cron_expr"` Timezone string `yaml:"timezone" json:"timezone"` Prompt string `yaml:"prompt" json:"prompt"` - Enabled *bool `yaml:"enabled" json:"enabled"` + // PromptFile is a file ref alternative to inline Prompt. Path is + // resolved relative to the workspace's files_dir. Inline Prompt wins + // when both are set. Scalability: hourly/weekly cron prompts are the + // largest text bodies in org.yaml (~1-5 KB each); externalizing them + // cuts the file by ~62%. + PromptFile string `yaml:"prompt_file" json:"prompt_file"` + Enabled *bool `yaml:"enabled" json:"enabled"` } // OrgChannel defines a social channel (Telegram, Slack, etc.) to auto-link @@ -112,32 +133,52 @@ type OrgChannel struct { } type OrgWorkspace struct { - Name string `yaml:"name" json:"name"` - Role string `yaml:"role" json:"role"` - Runtime string `yaml:"runtime" json:"runtime"` - Tier int `yaml:"tier" json:"tier"` - Template string `yaml:"template" json:"template"` - FilesDir string `yaml:"files_dir" json:"files_dir"` - SystemPrompt string `yaml:"system_prompt" json:"system_prompt"` - Model string `yaml:"model" json:"model"` - WorkspaceDir string `yaml:"workspace_dir" json:"workspace_dir"` - WorkspaceAccess string `yaml:"workspace_access" json:"workspace_access"` // #65: "none" (default), "read_only", "read_write" - Plugins []string `yaml:"plugins" json:"plugins"` - InitialPrompt string `yaml:"initial_prompt" json:"initial_prompt"` + Name string `yaml:"name" json:"name"` + Role string `yaml:"role" json:"role"` + Runtime string `yaml:"runtime" json:"runtime"` + Tier int `yaml:"tier" json:"tier"` + Template string `yaml:"template" json:"template"` + FilesDir string `yaml:"files_dir" json:"files_dir"` + // SystemPrompt is an inline override. Normally each role's system-prompt.md + // lives at `/system-prompt.md` and is copied via the files_dir + // template-copy step; inline overrides that path for ad-hoc workspaces. + SystemPrompt string `yaml:"system_prompt" json:"system_prompt"` + Model string `yaml:"model" json:"model"` + WorkspaceDir string `yaml:"workspace_dir" json:"workspace_dir"` + WorkspaceAccess string `yaml:"workspace_access" json:"workspace_access"` // #65: "none" (default), "read_only", "read_write" + Plugins []string `yaml:"plugins" json:"plugins"` + // InitialPrompt is the one-shot boot prompt. Agents run this once on first + // start; the body often clones the repo, reads CLAUDE.md + system-prompt, + // and commits conventions to memory. InitialPromptFile is the file-ref + // alternative — read at import time from `/`. + // Inline wins when both are set. + InitialPrompt string `yaml:"initial_prompt" json:"initial_prompt"` + InitialPromptFile string `yaml:"initial_prompt_file" json:"initial_prompt_file"` + // IdlePrompt / IdleIntervalSeconds drive the idle-loop reflection + // pattern (#205). When IdlePrompt is non-empty, the workspace self-sends + // this prompt every IdleIntervalSeconds while heartbeat.active_tasks == 0. + // Both fields were previously dropped by the org importer (struct didn't + // declare them); Phase 1 scalability PR adds them so engineer + researcher + // idle loops propagate correctly from org.yaml → /configs/config.yaml. + // IdlePromptFile is the file-ref alternative — same semantics as + // InitialPromptFile. Inline wins when both are set. + IdlePrompt string `yaml:"idle_prompt" json:"idle_prompt"` + IdlePromptFile string `yaml:"idle_prompt_file" json:"idle_prompt_file"` + IdleIntervalSeconds int `yaml:"idle_interval_seconds" json:"idle_interval_seconds"` // CategoryRouting extends/overrides defaults.category_routing per-workspace. // Merge semantics: workspace keys replace defaults' value for the same key // (empty list drops the category entirely); new keys are added. See // mergeCategoryRouting. CategoryRouting map[string][]string `yaml:"category_routing" json:"category_routing"` - Schedules []OrgSchedule `yaml:"schedules" json:"schedules"` - Channels []OrgChannel `yaml:"channels" json:"channels"` - External bool `yaml:"external" json:"external"` - URL string `yaml:"url" json:"url"` - Canvas struct { + Schedules []OrgSchedule `yaml:"schedules" json:"schedules"` + Channels []OrgChannel `yaml:"channels" json:"channels"` + External bool `yaml:"external" json:"external"` + URL string `yaml:"url" json:"url"` + Canvas struct { X float64 `yaml:"x" json:"x"` Y float64 `yaml:"y" json:"y"` } `yaml:"canvas" json:"canvas"` - Children []OrgWorkspace `yaml:"children" json:"children"` + Children []OrgWorkspace `yaml:"children" json:"children"` } // ListTemplates handles GET /org/templates — lists available org templates. @@ -437,10 +478,21 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, defa } } - // Inject initial_prompt into config.yaml (workspace-level overrides default) - initialPrompt := ws.InitialPrompt + // Resolve initial_prompt — inline wins, then file-ref, then defaults + // (inline → file → defaults.inline → defaults.file). File refs are + // rooted at // per resolvePromptRef semantics. + initialPrompt, err := resolvePromptRef(ws.InitialPrompt, ws.InitialPromptFile, orgBaseDir, ws.FilesDir) + if err != nil { + log.Printf("Org import: failed to resolve initial_prompt for %s: %v", ws.Name, err) + } if initialPrompt == "" { - initialPrompt = defaults.InitialPrompt + // Fall back to defaults. Defaults live at the org root, so they + // resolve with empty filesDir (relative to orgBaseDir). + var defaultErr error + initialPrompt, defaultErr = resolvePromptRef(defaults.InitialPrompt, defaults.InitialPromptFile, orgBaseDir, "") + if defaultErr != nil { + log.Printf("Org import: failed to resolve defaults.initial_prompt for %s: %v", ws.Name, defaultErr) + } } if initialPrompt != "" { if configFiles == nil { @@ -458,6 +510,46 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, defa log.Printf("Org import: injected initial_prompt (%d chars) into config.yaml for %s", len(trimmed), ws.Name) } + // Resolve idle_prompt — same precedence (ws inline → ws file → defaults). + // Inject into config.yaml alongside idle_interval_seconds so the + // workspace's heartbeat loop picks up the idle-reflection cadence on + // boot (see workspace-template/heartbeat.py + config.py). + idlePrompt, err := resolvePromptRef(ws.IdlePrompt, ws.IdlePromptFile, orgBaseDir, ws.FilesDir) + if err != nil { + log.Printf("Org import: failed to resolve idle_prompt for %s: %v", ws.Name, err) + } + if idlePrompt == "" { + var defaultErr error + idlePrompt, defaultErr = resolvePromptRef(defaults.IdlePrompt, defaults.IdlePromptFile, orgBaseDir, "") + if defaultErr != nil { + log.Printf("Org import: failed to resolve defaults.idle_prompt for %s: %v", ws.Name, defaultErr) + } + } + idleInterval := ws.IdleIntervalSeconds + if idleInterval == 0 { + idleInterval = defaults.IdleIntervalSeconds + } + if idlePrompt != "" { + if configFiles == nil { + configFiles = map[string][]byte{} + } + trimmed := strings.TrimSpace(idlePrompt) + lines := strings.Split(trimmed, "\n") + for i, line := range lines { + lines[i] = strings.TrimRight(line, " \t") + } + indented := strings.Join(lines, "\n ") + // idle_interval_seconds belongs with idle_prompt — empty idle_prompt + // means the idle loop never fires regardless of interval, so we + // only emit interval when there's a body to go with it. + if idleInterval <= 0 { + idleInterval = 600 // same default as workspace-template/config.py + } + block := fmt.Sprintf("idle_interval_seconds: %d\nidle_prompt: |\n %s\n", idleInterval, indented) + configFiles["config.yaml"] = appendYAMLBlock(configFiles["config.yaml"], block) + log.Printf("Org import: injected idle_prompt (%d chars, interval=%ds) into config.yaml for %s", len(trimmed), idleInterval, ws.Name) + } + // Inline system_prompt (only if no files_dir provides one) if ws.SystemPrompt != "" { if configFiles == nil { @@ -503,7 +595,10 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, defa go h.workspace.provisionWorkspace(id, templatePath, configFiles, payload) } - // Insert schedules if defined + // Insert schedules if defined. Resolve each schedule's prompt body from + // either inline `prompt:` or `prompt_file:` (file ref relative to the + // workspace's files_dir). Inline wins; empty prompt after resolution is + // a configuration error (cron with no body would never do anything). for _, sched := range ws.Schedules { tz := sched.Timezone if tz == "" { @@ -513,12 +608,21 @@ func (h *OrgHandler) createWorkspaceTree(ws OrgWorkspace, parentID *string, defa if sched.Enabled != nil { enabled = *sched.Enabled } + prompt, promptErr := resolvePromptRef(sched.Prompt, sched.PromptFile, orgBaseDir, ws.FilesDir) + if promptErr != nil { + log.Printf("Org import: failed to resolve prompt for schedule '%s' on %s: %v — skipping insert", sched.Name, ws.Name, promptErr) + continue + } + if prompt == "" { + log.Printf("Org import: schedule '%s' on %s has empty prompt (neither prompt nor prompt_file set) — skipping insert", sched.Name, ws.Name) + continue + } nextRun, _ := scheduler.ComputeNextRun(sched.CronExpr, tz, time.Now()) if _, err := db.DB.ExecContext(context.Background(), orgImportScheduleSQL, - id, sched.Name, sched.CronExpr, tz, sched.Prompt, enabled, nextRun); err != nil { + id, sched.Name, sched.CronExpr, tz, prompt, enabled, nextRun); err != nil { log.Printf("Org import: failed to upsert schedule '%s' for %s: %v", sched.Name, ws.Name, err) } else { - log.Printf("Org import: schedule '%s' (%s) upserted for %s (source=template)", sched.Name, sched.CronExpr, ws.Name) + log.Printf("Org import: schedule '%s' (%s, %d chars) upserted for %s (source=template)", sched.Name, sched.CronExpr, len(prompt), ws.Name) } } @@ -638,6 +742,53 @@ func countWorkspaces(workspaces []OrgWorkspace) int { return count } +// resolvePromptRef reads a prompt body from either an inline string or a +// file ref relative to the workspace's files_dir. Inline always wins when +// both are non-empty (caller-provided inline is more authoritative than a +// file path that may not exist yet during dev loops). +// +// File resolution: +// - `//` when filesDir is non-empty +// - `/` when filesDir is empty (defaults-level refs) +// +// Both paths go through resolveInsideRoot so a crafted fileRef can't escape +// the org template directory via traversal (same defense the files_dir +// copy-step uses). +// +// Returns (resolved body, error). If both inline and fileRef are empty, +// returns ("", nil) — caller decides whether that's a problem. +func resolvePromptRef(inline, fileRef, orgBaseDir, filesDir string) (string, error) { + if inline != "" { + return inline, nil + } + if fileRef == "" { + return "", nil + } + if orgBaseDir == "" { + // Inline-only template (POST /org/import with a raw Template in the + // JSON body, not a dir). File refs can't be resolved — surface the + // problem rather than silently returning empty. + return "", fmt.Errorf("prompt_file %q requires a dir-based org template (no orgBaseDir in inline-template mode)", fileRef) + } + searchRoot := orgBaseDir + if filesDir != "" { + p, err := resolveInsideRoot(orgBaseDir, filesDir) + if err != nil { + return "", fmt.Errorf("invalid files_dir %q: %w", filesDir, err) + } + searchRoot = p + } + abs, err := resolveInsideRoot(searchRoot, fileRef) + if err != nil { + return "", fmt.Errorf("invalid prompt_file %q: %w", fileRef, err) + } + data, err := os.ReadFile(abs) + if err != nil { + return "", fmt.Errorf("read prompt_file %q: %w", fileRef, err) + } + return string(data), nil +} + // envVarRefPattern matches actual ${VAR} or $VAR references (not literal $). // Used to detect unresolved placeholders without false positives like "$5". var envVarRefPattern = regexp.MustCompile(`\$\{?[A-Za-z_][A-Za-z0-9_]*\}?`) diff --git a/platform/internal/handlers/org_prompt_ref_test.go b/platform/internal/handlers/org_prompt_ref_test.go new file mode 100644 index 00000000..3141ec8f --- /dev/null +++ b/platform/internal/handlers/org_prompt_ref_test.go @@ -0,0 +1,221 @@ +package handlers + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +// YAML parsing coverage for the new phase-1 scalability fields. +// Catches regressions where someone renames a struct tag and the YAML +// loader silently drops the value (what the prior idle_prompt bug was — +// the struct simply didn't declare the field, so org.yaml entries were +// dropped on import). + +func TestOrgYAML_IdlePromptFieldsParse(t *testing.T) { + src := ` +name: Test Org +workspaces: + - name: Role A + files_dir: role-a + idle_prompt: "inline idle body" + idle_interval_seconds: 300 + - name: Role B + files_dir: role-b + idle_prompt_file: idle-prompt.md + idle_interval_seconds: 600 +` + var tmpl OrgTemplate + if err := yaml.Unmarshal([]byte(src), &tmpl); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + if len(tmpl.Workspaces) != 2 { + t.Fatalf("expected 2 workspaces, got %d", len(tmpl.Workspaces)) + } + a := tmpl.Workspaces[0] + if a.IdlePrompt != "inline idle body" { + t.Errorf("idle_prompt inline not parsed: %q", a.IdlePrompt) + } + if a.IdleIntervalSeconds != 300 { + t.Errorf("idle_interval_seconds not parsed: %d", a.IdleIntervalSeconds) + } + b := tmpl.Workspaces[1] + if b.IdlePromptFile != "idle-prompt.md" { + t.Errorf("idle_prompt_file not parsed: %q", b.IdlePromptFile) + } + if b.IdleIntervalSeconds != 600 { + t.Errorf("idle_interval_seconds not parsed: %d", b.IdleIntervalSeconds) + } +} + +func TestOrgYAML_PromptFileFieldsParse(t *testing.T) { + src := ` +name: Test Org +defaults: + initial_prompt_file: shared-boot.md +workspaces: + - name: Role A + files_dir: role-a + initial_prompt_file: initial-prompt.md + schedules: + - name: Hourly audit + cron_expr: "17 * * * *" + prompt_file: schedules/hourly-audit.md +` + var tmpl OrgTemplate + if err := yaml.Unmarshal([]byte(src), &tmpl); err != nil { + t.Fatalf("unmarshal failed: %v", err) + } + if tmpl.Defaults.InitialPromptFile != "shared-boot.md" { + t.Errorf("defaults.initial_prompt_file not parsed: %q", tmpl.Defaults.InitialPromptFile) + } + if len(tmpl.Workspaces) != 1 { + t.Fatalf("expected 1 workspace, got %d", len(tmpl.Workspaces)) + } + w := tmpl.Workspaces[0] + if w.InitialPromptFile != "initial-prompt.md" { + t.Errorf("initial_prompt_file not parsed: %q", w.InitialPromptFile) + } + if len(w.Schedules) != 1 { + t.Fatalf("expected 1 schedule, got %d", len(w.Schedules)) + } + if w.Schedules[0].PromptFile != "schedules/hourly-audit.md" { + t.Errorf("schedule.prompt_file not parsed: %q", w.Schedules[0].PromptFile) + } +} + +// resolvePromptRef is the single entry point for reading workspace prompt +// bodies from either inline YAML or sibling .md files. Phase 1 of the +// org.yaml externalization work (1801-line file → ~600-line structural +// manifest). Tests cover the 6 cases callers exercise + the traversal +// defense (same class as resolveInsideRoot). + +func TestResolvePromptRef_InlineWinsOverFile(t *testing.T) { + tmp := t.TempDir() + rolesDir := filepath.Join(tmp, "my-role") + if err := os.MkdirAll(rolesDir, 0o755); err != nil { + t.Fatal(err) + } + // Write a file that would be read if inline were empty — prove inline wins. + if err := os.WriteFile(filepath.Join(rolesDir, "idle.md"), []byte("FROM FILE"), 0o644); err != nil { + t.Fatal(err) + } + got, err := resolvePromptRef("FROM INLINE", "idle.md", tmp, "my-role") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != "FROM INLINE" { + t.Errorf("inline should win: got %q", got) + } +} + +func TestResolvePromptRef_FileReadWhenInlineEmpty(t *testing.T) { + tmp := t.TempDir() + rolesDir := filepath.Join(tmp, "my-role") + if err := os.MkdirAll(rolesDir, 0o755); err != nil { + t.Fatal(err) + } + want := "You have no active task.\nBacklog-pull + reflect." + if err := os.WriteFile(filepath.Join(rolesDir, "idle.md"), []byte(want), 0o644); err != nil { + t.Fatal(err) + } + got, err := resolvePromptRef("", "idle.md", tmp, "my-role") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != want { + t.Errorf("got %q, want %q", got, want) + } +} + +func TestResolvePromptRef_BothEmptyReturnsEmpty(t *testing.T) { + tmp := t.TempDir() + got, err := resolvePromptRef("", "", tmp, "any") + if err != nil { + t.Errorf("empty inputs should not error, got: %v", err) + } + if got != "" { + t.Errorf("empty inputs should return empty body, got %q", got) + } +} + +func TestResolvePromptRef_DefaultsLevelResolvesRelativeToOrgBase(t *testing.T) { + // Defaults don't have a files_dir — ref is resolved relative to + // orgBaseDir itself (shared prompts at the org root). + tmp := t.TempDir() + if err := os.WriteFile(filepath.Join(tmp, "shared.md"), []byte("SHARED"), 0o644); err != nil { + t.Fatal(err) + } + got, err := resolvePromptRef("", "shared.md", tmp, "") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != "SHARED" { + t.Errorf("got %q, want SHARED", got) + } +} + +func TestResolvePromptRef_InlineTemplateErrors(t *testing.T) { + // When the caller used inline-template mode (POST /org/import with a + // raw Template in the JSON body, no dir), orgBaseDir is empty and file + // refs are unresolvable. Surface that loudly. + _, err := resolvePromptRef("", "idle.md", "", "my-role") + if err == nil { + t.Error("expected error for file ref without orgBaseDir") + } + if !strings.Contains(err.Error(), "inline-template") { + t.Errorf("error should mention inline-template mode; got: %v", err) + } +} + +func TestResolvePromptRef_RejectsTraversalViaFileRef(t *testing.T) { + tmp := t.TempDir() + rolesDir := filepath.Join(tmp, "my-role") + if err := os.MkdirAll(rolesDir, 0o755); err != nil { + t.Fatal(err) + } + // Create a sensitive file OUTSIDE the orgBaseDir we can try to exfiltrate. + outside := filepath.Join(filepath.Dir(tmp), "secret.md") + if err := os.WriteFile(outside, []byte("SECRET"), 0o644); err != nil { + t.Fatal(err) + } + defer os.Remove(outside) + + cases := []string{ + "../../secret.md", + "../secret.md", + "../../../../etc/passwd", + } + for _, tc := range cases { + t.Run(tc, func(t *testing.T) { + _, err := resolvePromptRef("", tc, tmp, "my-role") + if err == nil { + t.Errorf("expected error for traversal %q", tc) + } + }) + } +} + +func TestResolvePromptRef_RejectsTraversalViaFilesDir(t *testing.T) { + tmp := t.TempDir() + // The files_dir argument also comes from YAML — ensure it can't escape. + _, err := resolvePromptRef("", "idle.md", tmp, "../escape") + if err == nil { + t.Error("expected error for traversal via files_dir") + } +} + +func TestResolvePromptRef_MissingFileErrors(t *testing.T) { + tmp := t.TempDir() + rolesDir := filepath.Join(tmp, "my-role") + if err := os.MkdirAll(rolesDir, 0o755); err != nil { + t.Fatal(err) + } + _, err := resolvePromptRef("", "nonexistent.md", tmp, "my-role") + if err == nil { + t.Error("expected error for missing file") + } +}