content hashing, memory versioning, memory decay

[ishaanxgupta]

Add content hashing: SHA-256 hash of content for deduplication — skip re-processing if the hash matches.
Add memory versioning: Track memory lineage (parent_memory_id, version) so updates don't lose history.
Add Proper forget Mechanism

Bounty: 30$ API Credits

[ishaanxgupta]

Please discuss the proper plan here first then only raise a PR

[kodahhhhh]

Plan for this:\n\n1. Add SHA-256 content hashes to vector-memory metadata and check that hash before embedding/writing, so duplicate content can be skipped without re-processing.\n2. Add memory lineage metadata with parent_memory_id, version, and is_current. New memories start at version 1; updates create a new current vector version and mark the previous vector as superseded instead of overwriting history.\n3. Make retrieval/judge paths ignore superseded/forgotten vector memories, while still allowing legacy records without is_current metadata to be read.\n4. Make DELETE/forget remove every vector in the same lineage, so forgetting the current version also removes older superseded versions.\n5. Cover this with focused Weaver tests for duplicate-hash skips, versioned updates, and full-lineage forgets.

[RYB-404]

/attempt #166

Plan before PR, per maintainer request:

1. Add deterministic SHA-256 content hashing at the memory write boundary, normalizing text before hashing so repeated writes of the same content are idempotent and skip embedding/vector writes.
2. Store explicit lineage metadata on each memory record: content_hash, parent_memory_id, version, is_current, forgotten_at, and forget_reason where supported by the existing storage layer.
3. Implement updates as append-only versions: new version becomes current, prior versions are retained but marked non-current so history is preserved without polluting retrieval.
4. Implement a proper forget path that excludes forgotten memories from search/retrieval immediately, with a hard-delete option only if the current store already supports physical deletion cleanly.
5. Add focused regression tests for duplicate-content skip, version lineage/current switching, and forgotten memories being excluded from retrieval.

I will keep the implementation aligned with the existing XMem storage abstractions and avoid broad rewrites.

[chiragkhatri19]

Should I give it a shot?

[chiragkhatri19]

Hi @ishaanxgupta, sharing a plan before opening a PR per your request.

Credit first: @kodahhhhh's #187 nailed store-agnosticism via BaseVectorStore, and @RYB-404's #199 has a clean dedicated lifecycle module and soft-forget. I would like to build on both by taking #187's backend-agnosticism and #199's clean module, closing the two open review issues, and adding the decay the title asks for (which neither PR covers yet).

Architecture: Lifecycle at the store boundary, not the core.
Pure logic lives in a new src/pipelines/lifecycle.py (so it is inside the coverage gate) alongside a Pydantic MemoryLifecycle schema. Weaver delegates dedup, supersede, and forget functions to it, while retrieval calls one filter and decay function. There are no judge.py edits, no lifecycle branching in the deterministic core, and no per-backend code. This works uniformly on SQLite, Chroma, PGVector, and Pinecone through BaseVectorStore.

Closes the full class of lifecycle-field bugs in #199 structurally — not just the update() P1, but also the setdefault silent-data-loss footgun (a caller passing
{"is_current": False} to add() gets an invisible record with no error) and the SQLite IS NOT 0 = NULL edge case (legacy records without is_current in metadata matching the
dedup filter). All three stem from the same root: lifecycle fields being controllable by callers. Our fix: every write path strips all reserved keys from caller metadata
first, then re-pins authoritative values — so the fields are never caller-settable, regardless of which write path or backend is used.

The Four Requirements:

• Hashing: Reuse the existing stable_hash (src/jobs/durable.py), normalize by stripping, applying NFC, and collapsing whitespace, and scope by user_id, domain, and namespace (keys persisted today, meaning no cross-user reuse). Deduplication runs inside the batch flush (the real default path via _execute_batched_vector), executing one search_by_metadata per flush rather than one per item. On a hit, it skips embed and write, reusing the ID. Beyond a cap, it degrades to best-effort where a miss simply stores a duplicate and never corrupts a lineage.
• Versioning: Introduce parent_memory_id, version, and is_current. Superseding skips re-embedding when the content hash is unchanged (this is a metadata-only update where backends keep the old embedding on embedding=None). It uses new-version-then-flip-old ordering. Readers resolve by max(version) among is_current records, ensuring it never resolves to two-current without a cross-store transaction.
• Forget: Actions are soft by default (lifecycle_state="forgotten", forgotten_at, forget_reason), hiding them from retrieval but keeping them in get(). It also includes hard=True to delete the whole lineage.
• Decay: This is deterministic and uses an injectable clock, but remains OFF by default behind a config flag. It applies the following formula to summary results and re-sorts them so it actually affects ranking:
  $$final=base\cdot\left(floor+(1-floor)\cdot0.5^{\frac{age_days}{half_life_days}}\right)$$
  With the flag off, it yields byte-identical behavior, leaving profile/temporal and the Neo4j threshold untouched.

Backward compatible & gated.
This introduces additive metadata only, with all fields defaulted. Legacy records are treated as current and retrievable with no API-response changes. Tests run on the in-memory fake, including regressions that would have caught #199's resurrection hole and the cross-user deduplication bug, plus a fix for a latent fake bug where it replaced metadata while real backends merge.

Footprint: Adds 2 new files and thin seams in weaver.py, retrieval.py, and settings.py. This is one PR with one concern, keeping decay flag-gated independently. Multi-tenant scope-key plumbing is a deliberate follow-up.

Does this direction look right, particularly (a) keeping the lifecycle at the store boundary vs the core, and (b) defaulting decay to off behind a flag? I am happy to adjust scope before I open the PR. I would like to claim the issue if so.

[ishaanxgupta]

Hi @chiragkhatri19, approach looks correct mostly I would love you to make these changes in the /v2 routes only. lets start with forget feature first. just adding the metadata forget = true/false is fine for now during ingestion. We will see how we need to use this and in what plugins we can keep the forget = true. If forget = true, you will have to add a TTL alongwith it. Rest your approach looks good for forgetting.
for versioning, like should we create parents when the memory gets updated, because in our whole pipeline, we loose the data only when update happens by the weaver. so at that point I think it would be really great to create versions of the memory. what do you think?
for decay we will have to think of a formula for this and at some point this has to be merged with the forget thing. You can skip that for now.