molecule-ai/molecule-core
52
0
Fork 2
Code Issues 179 Pull Requests 59 Actions 4 Packages Projects Releases Wiki Activity

memory: Phase A2 — backfill agent_memories rows into v2 plugin per tenant #1791

New Issue
Closed
opened 2026-05-24 09:13:21 +00:00 by hongming · 1 comment
hongming commented 2026-05-24 09:13:21 +00:00
Owner
Copy Link
Copy Source

Summary

Phase A2 of the v1→v2 memory migration. Phase A1 (#1747 — kill the v1 SQL fallback) is live in production as of 2026-05-24; the v2 plugin is the only memory backend on every active tenant. A2 is the per-tenant cleanup: existing agent_memories rows need to be migrated into the v2 plugin so historical memory isn't orphaned.

Why now

  • Phase A1 verified stable in production (all 4 active tenants on platform-tenant:staging-272cb8b since 2026-05-24 08:05Z, memory plugin sidecar healthy, MEMORY_PLUGIN_URL wired).
  • Every memory write post-A1 goes to v2. Every read goes to v2. The agent_memories table is now read-only legacy state.
  • Without A2, anything an agent wrote pre-A1 is invisible — UI reads from v2, the row is in v1.

Scope

For each active tenant (currently 4: agents-team, hongming, chloe-dong, reno-stars):

  1. Read all rows from agent_memories on the tenant's DB.
  2. For each row, call the local memory plugin's CommitMemory with:
    • namespace = workspace:<workspace_id> (matches workspaceMemoryNamespace())
    • content = agent_memories.content
    • kind = MemoryKindFact (legacy rows are pre-commit_memory_v2; treat as factual baseline)
    • source = MemorySourceAgent (was written by agent in the old MCP path)
  3. Mark the source row as migrated_at = now() (additive column, no row delete) so A3 can verify completeness later.
  4. Idempotent: re-running the script must be a no-op (check migrated_at).

Implementation hints

  • Script lives in workspace-server/cmd/memory-plugin-postgres/migrations/ or as a one-shot in scripts/ops/.
  • Best run via SSM on the tenant EC2 — already-authenticated path used by RedeployTenant.
  • Per-tenant run is independent (each tenant has its own DB + plugin).
  • Per memory feedback_per_agent_gitea_identity_default — the migration script should run under a tenant-scoped service identity, not the founder PAT.

Acceptance

  • Script lives in repo with tests against sqlmock covering: empty source table → no-op; populated source → all rows committed; pre-migrated rows → skip.
  • Dry-run mode (--check-only) reports counts without writing.
  • Run against all 4 active tenants. Report per-tenant counts (rows migrated, rows skipped, rows failed).
  • Spot-check: pick 3 rows on a tenant, verify they're now visible via recall_memory MCP call AND via the canvas MemoryInspectorPanel.

Gating for Phase A3

Once this lands and a 30-day soak passes with no v1-vs-v2 divergence reports, file Phase A3 to drop the agent_memories table.

Discovered during

Multi-PR memory-system migration 2026-05-24. Phase A1 = #1747. Phase A2 = this issue. Phase A3 = follow-on.

## Summary Phase A2 of the v1→v2 memory migration. Phase A1 (#1747 — kill the v1 SQL fallback) is live in production as of 2026-05-24; the v2 plugin is the only memory backend on every active tenant. A2 is the per-tenant cleanup: existing `agent_memories` rows need to be migrated into the v2 plugin so historical memory isn't orphaned. ## Why now - Phase A1 verified stable in production (all 4 active tenants on `platform-tenant:staging-272cb8b` since 2026-05-24 08:05Z, memory plugin sidecar healthy, MEMORY_PLUGIN_URL wired). - Every memory write post-A1 goes to v2. Every read goes to v2. The `agent_memories` table is now read-only legacy state. - Without A2, anything an agent wrote pre-A1 is invisible — UI reads from v2, the row is in v1. ## Scope For each active tenant (currently 4: `agents-team`, `hongming`, `chloe-dong`, `reno-stars`): 1. Read all rows from `agent_memories` on the tenant's DB. 2. For each row, call the local memory plugin's `CommitMemory` with: - namespace = `workspace:<workspace_id>` (matches `workspaceMemoryNamespace()`) - content = `agent_memories.content` - kind = `MemoryKindFact` (legacy rows are pre-`commit_memory_v2`; treat as factual baseline) - source = `MemorySourceAgent` (was written by agent in the old MCP path) 3. Mark the source row as `migrated_at = now()` (additive column, no row delete) so A3 can verify completeness later. 4. Idempotent: re-running the script must be a no-op (check `migrated_at`). ## Implementation hints - Script lives in `workspace-server/cmd/memory-plugin-postgres/migrations/` or as a one-shot in `scripts/ops/`. - Best run via SSM on the tenant EC2 — already-authenticated path used by RedeployTenant. - Per-tenant run is independent (each tenant has its own DB + plugin). - Per memory `feedback_per_agent_gitea_identity_default` — the migration script should run under a tenant-scoped service identity, not the founder PAT. ## Acceptance - [ ] Script lives in repo with tests against `sqlmock` covering: empty source table → no-op; populated source → all rows committed; pre-migrated rows → skip. - [ ] Dry-run mode (`--check-only`) reports counts without writing. - [ ] Run against all 4 active tenants. Report per-tenant counts (rows migrated, rows skipped, rows failed). - [ ] Spot-check: pick 3 rows on a tenant, verify they're now visible via `recall_memory` MCP call AND via the canvas MemoryInspectorPanel. ## Gating for Phase A3 Once this lands and a 30-day soak passes with no v1-vs-v2 divergence reports, file Phase A3 to drop the `agent_memories` table. ## Discovered during Multi-PR memory-system migration 2026-05-24. Phase A1 = #1747. Phase A2 = this issue. Phase A3 = follow-on.
hongming commented 2026-05-24 11:05:00 +00:00
Author
Owner
Copy Link
Copy Source

Phase A2 complete — closing

End-to-end verification on production (2026-05-24):

Tenant agent_memories (v1) memory_plugin.memory_records (v2) Status
agents-team 1805 (frozen) 1805 → 1807 (live writes) ✅
hongming 144 (frozen) 144 ✅
chloe-dong 1 (frozen) 1 ✅
reno-stars 102 (frozen) 102 ✅

Total: 2,052 historical rows migrated, zero errors. Live POST /memories now returns 201 with the row landing in memory_plugin.memory_records. The v2 plugin row count grew by 2 immediately post-backfill, confirming that the writer migration (#1794) is taking effect — new writes go to v2 only, agent_memories is frozen.

Chain summary

PR Role
#1794 Route POST /memories through v2 plugin (the writer migration)
#1795 Broadcast ACTIVITY_LOGGED on MCP memory writes (#1754 fix)
#1796 Bundle memory-backfill CLI into tenant image
#1798 URGENT fix — marshalMetadata nil→null (caught latent pq+jsonb bug that #1794 unmasked in production)

Post-recycle on staging-7604e11:

  1. All 4 tenants verified on new image with /memory-backfill binary
  2. Backfill applied per-tenant via docker exec molecule-tenant /memory-backfill -apply
  3. Live writes verified going to v2

Closes

Closes #1791 (Phase A2). Phase A3 (#1792 — drop agent_memories table) remains open per its 30-day soak gate.

## Phase A2 complete — closing End-to-end verification on production (2026-05-24): | Tenant | agent_memories (v1) | memory_plugin.memory_records (v2) | Status | |---|---|---|---| | agents-team | 1805 (frozen) | 1805 → 1807 (live writes) | ✅ | | hongming | 144 (frozen) | 144 | ✅ | | chloe-dong | 1 (frozen) | 1 | ✅ | | reno-stars | 102 (frozen) | 102 | ✅ | **Total: 2,052 historical rows migrated, zero errors.** Live POST /memories now returns 201 with the row landing in `memory_plugin.memory_records`. The v2 plugin row count grew by 2 immediately post-backfill, confirming that the writer migration (#1794) is taking effect — new writes go to v2 only, agent_memories is frozen. ### Chain summary | PR | Role | |---|---| | #1794 | Route POST /memories through v2 plugin (the writer migration) | | #1795 | Broadcast ACTIVITY_LOGGED on MCP memory writes (#1754 fix) | | #1796 | Bundle memory-backfill CLI into tenant image | | **#1798** | URGENT fix — marshalMetadata nil→`null` (caught latent pq+jsonb bug that #1794 unmasked in production) | Post-recycle on `staging-7604e11`: 1. All 4 tenants verified on new image with /memory-backfill binary 2. Backfill applied per-tenant via `docker exec molecule-tenant /memory-backfill -apply` 3. Live writes verified going to v2 ### Closes Closes #1791 (Phase A2). Phase A3 (#1792 — drop `agent_memories` table) remains open per its 30-day soak gate.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1791
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?