Self-review missed deliverable from PR-7's task spec. Operators had
no way to confirm a -apply produced equivalent search results to the
legacy agent_memories direct queries; this PR ships that.
Usage:
memory-backfill -verify # 50-workspace random sample
memory-backfill -verify -verify-sample=200 # bigger sample
memory-backfill -verify -workspace=<uuid> # one specific workspace
Algorithm:
1. Pick N random workspaces (or use -workspace if specified)
2. For each: query agent_memories direct, query plugin search via
the workspace's readable namespace list
3. Multiset-compare contents: every legacy row must have a matching
plugin row. Plugin having MORE rows is OK (team-shared content
may be visible from sibling workspaces).
4. Print mismatches with content excerpt; non-zero mismatches/errors
yields a non-zero exit so CI can gate cutover.
Sql:
- Sampling uses ORDER BY random() LIMIT N (TABLESAMPLE has surprising
distribution at small populations).
- Filters out status='removed' workspaces.
Test coverage:
* pickWorkspaceSample: single-ws short-circuit, random sampling,
query error, scan error
* queryLegacyMemories: happy path, error path
* verifyParity:
- all match → 1 match, 0 mismatch
- missing-from-plugin → 1 mismatch with content excerpt
- plugin-extra rows → 1 match (legacy is subset of plugin)
- legacy query error → 1 error counter
- resolver error → 1 error counter
- plugin search error → 1 error counter
- no readable namespaces + empty legacy → match
- no readable namespaces + non-empty legacy → mismatch
- pickSample error → propagated up
* CLI: -verify+-apply rejected as mutually exclusive; -verify alone
is a valid mode
Note: namespaceResolverAdapter bridges *namespace.Resolver to the
verify package's verifyResolver interface so verify.go has zero
dependency on the namespace package — keeps test stubs minimal.
Self-review (post-merge) flagged that the backfill claimed to be
idempotent on re-run but actually duplicates every row because the
plugin's INSERT uses gen_random_uuid() and ignores any id passed in.
Fix is contract-level: extend MemoryWrite with an optional `id`
idempotency key. When supplied, the plugin MUST treat the write as
upsert keyed on this id; when omitted, the plugin generates a fresh
UUID (production agent commits keep working unchanged).
Changes:
* docs/api-protocol/memory-plugin-v1.yaml: add id field with
description that flags it as idempotency key
* internal/memory/contract/contract.go: add ID to MemoryWrite struct,
update memory_write_minimal golden vector
* internal/memory/pgplugin/store.go: split CommitMemory into two
paths — upsert when body.ID set (INSERT ... ON CONFLICT (id) DO
UPDATE), plain INSERT otherwise
* cmd/memory-backfill/main.go: pass agent_memories.id to MemoryWrite,
fix the false comment about 409 deduplication
New tests:
* pgplugin: TestCommitMemory_WithIDUpserts pins the upsert SQL is
used when id is set; TestCommitMemory_UpsertScanError covers the
error branch
* backfill: TestBackfill_PassesSourceUUIDAsIdempotencyKey pins the
forwarding behavior; TestBackfill_RerunIsIdempotent simulates a
retry and asserts both runs pass the same uuid (plugin upsert is
what makes this safe)
Why this matters: operators retrying a failed backfill (which they
will — networks fail, transactions abort) would otherwise create N
duplicates per memory. The duplicates aren't visible until search
results show obvious dupes — debugging that under prod load is bad.
Production agent commits are unaffected: they leave id empty, the
plugin generates a fresh UUID via gen_random_uuid(), zero behavior
change for the hot path.
Builds on merged PR-1..6. Operator runs this once at cutover to copy
agent_memories rows into the v2 plugin's storage.
Usage:
memory-backfill -dry-run # count + diff, no writes
memory-backfill -apply # actually copy
memory-backfill -apply -limit=10000 # cap rows per run
memory-backfill -apply -workspace=<uuid> # one workspace only
Required env: DATABASE_URL + MEMORY_PLUGIN_URL.
Translation matches the PR-6 legacy shim:
LOCAL → workspace:<workspace_id>
TEAM → team:<root_id> (resolved via the same namespace.Resolver
the runtime uses)
GLOBAL → org:<root_id>
Idempotent: each row is keyed by its UUID; re-running the backfill
does not duplicate writes (plugin handles deduplication).
What ships:
* cmd/memory-backfill/main.go: CLI entry, run() driver,
backfill() workhorse, mapScopeToNamespace + namespaceKindFromString
helpers
* main_test.go: 100% on the functional logic (mapScopeToNamespace,
namespaceKindFromString, backfill(), all CLI validation paths)
Coverage: 80.2% of statements. The 19.8% gap is main()'s body
(log.Fatalf — not unit-testable) and run()'s real-DB integration
(sql.Open + db.PingContext + new client/resolver — requires a live
postgres). Integration coverage for this path lives in PR-11
(E2E plugin-swap test).
Edge cases pinned (in functional logic):
* Every legacy scope → namespace mapping
* Unknown scope → skip with diagnostic, increment skipped counter
* Resolver error → propagate, abort run
* No-matching-kind in writable list → skip with error message
* Plugin UpsertNamespace error → increment errors, continue
* Plugin CommitMemory error → increment errors, continue
* Query error → propagate, abort
* Scan error → increment errors, continue
* Mid-iteration row error → propagate, abort
* Workspace filter passes through to SQL WHERE clause
* Dry-run mode never calls plugin
* CLI: rejects both/neither modes, missing env vars, bad flags
