Memory System

SQLite + FTS5 + sqlite-vec — small context, good memory

Database Schema (v28)

The core tables below are the memory foundation. Later migrations add more tables — conversation_summaries, skills/vec_skills, topics, users, auth_events, user_profiles, schedules, calendar_events/calendar_reminders, and the email_calendar_extractions dedup ledger. See src/memory/schema.rs and FEATURES.md for the full set.

messages Phase 1

Append-only raw conversation log. Never modified or deleted.

CREATE TABLE messages (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    channel_id  TEXT NOT NULL,      -- Matrix room ID
    sender      TEXT NOT NULL,      -- User ID or "assistant"
    role        TEXT NOT NULL,      -- "user" / "assistant"
    content     TEXT NOT NULL,
    created_at  TEXT NOT NULL       -- ISO 8601 UTC
);

memories Phase 1 Phase 2

Extracted facts. Mutable — updated on dedup, archived on decay/merge.

CREATE TABLE memories (
    id            INTEGER PRIMARY KEY AUTOINCREMENT,
    channel_id    TEXT,              -- Matrix room ID
    user_id       TEXT,              -- Matrix user ID (NULL for room/global scope)
    content       TEXT NOT NULL,     -- The fact itself
    category      TEXT NOT NULL,     -- fact | preference | event | episode | general
    importance    REAL NOT NULL,     -- 0.0-1.0 (identity=1.0, trivial=0.2)
    strength      REAL NOT NULL,     -- Ebbinghaus decay (starts at 1.0)
    access_count  INTEGER NOT NULL,  -- bumped on retrieval
    last_accessed TEXT,              -- reset on retrieval (decay clock)
    source_ids    TEXT,              -- JSON array of merged memory IDs
    archived_at   TEXT,              -- soft-delete timestamp
    scope         TEXT NOT NULL,     -- personal | room | global
    created_at    TEXT NOT NULL,
    updated_at    TEXT NOT NULL
);

Memory Scopes

Scope	user_id	Visible to	Example
`personal`	Set	Only that user	"Oz prefers dark mode"
`room`	NULL	Everyone in the room	"Team meeting is every Thursday"
`global`	NULL	Everyone everywhere	"SearXNG runs on a self-hosted host"

The extraction LLM classifies each fact's scope based on whether it's about a specific person, shared group knowledge, or infrastructure/technical facts.

vec_memories Phase 3

sqlite-vec virtual table. 768-dimensional embeddings from nomic-embed-text.

CREATE VIRTUAL TABLE vec_memories USING vec0(
    memory_id INTEGER PRIMARY KEY,  -- links to memories.id
    embedding float[768]
);

maintenance_state Phase 2

Per-channel progress tracking for the maintenance worker.

CREATE TABLE maintenance_state (
    channel_id                TEXT PRIMARY KEY,
    last_episode_message_id   INTEGER NOT NULL DEFAULT 0,
    last_consolidation        TEXT,
    last_decay                TEXT,
    last_dream                TEXT       -- reflective "dream" pass watermark (v24)
);

memory_links Phase 4

Relationship graph between memories. Directed edges queried both ways.

CREATE TABLE memory_links (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    source_id  INTEGER NOT NULL,   -- references memories.id
    target_id  INTEGER NOT NULL,   -- references memories.id
    link_type  TEXT NOT NULL,      -- co_extracted | consolidated | semantic
    strength   REAL NOT NULL,      -- 1.0 for structural, cosine sim for semantic
    created_at TEXT NOT NULL,
    UNIQUE(source_id, target_id, link_type)
);

FTS5 Virtual Tables Phase 1

messages_fts and memories_fts — kept in sync via INSERT/DELETE/UPDATE triggers. Enables full-text search with BM25 ranking.

Extraction Worker

Background tokio task with a 64-slot mpsc channel. After each conversation exchange:

1. Identify the sender's display name from their Matrix user ID
2. Fetch related existing memories via FTS5 (scoped to this user)
3. LLM prompt: "extract facts about {username} from this exchange" (with existing memories for dedup context)
4. Parse JSON array of {content, category, importance, scope}
5. For each fact: Jaccard similarity check (threshold 0.35) against FTS5 candidates
6. If similar match: UPDATE existing memory. If new: INSERT with user_id and scope.
7. Generate embedding via Ollama and store in vec_memories

Facts are attributed by name ("Oz prefers..." not "the user prefers..."). Personal facts are tagged with the user's Matrix ID; room/global facts have no user_id. If the channel is full, extraction is silently dropped. The user never waits.

8. When multiple facts are extracted from the same exchange, they are linked with co_extracted edges in the memory graph.

Ebbinghaus Decay

Runs once per 24 hours during the maintenance cycle.

new_strength = strength × 2^{(-days_since_last_access / half_life)}

Default half-life: 14 days. A memory accessed yesterday barely decays. A memory untouched for 2 weeks drops to 50% strength. Below 10% → archived.

The touch_memories() call during retrieval resets last_accessed, implementing natural spaced repetition: frequently recalled memories stay strong.

Episodic Summaries

Every 30 minutes, the maintenance worker scans each channel for unsummarized messages.

If 6+ messages exist since the last episode watermark, they're grouped by 60-minute time windows. Each window of 4+ messages gets LLM-summarized into a 1-2 sentence "episode" memory.

Episodes reduce noise — instead of 20 individual facts from a long conversation, you get "Discussed GPU server setup: decided on 4x RTX 3090 config for the GPU server, tested with Gemma 4."

Fact Consolidation

Every 30 minutes, the maintenance worker finds clusters of similar memories using Jaccard word-overlap similarity (threshold 0.35).

Each cluster of 2+ similar memories gets LLM-merged into a single, comprehensive memory. The originals are soft-deleted (archived_at set), and the merged result tracks provenance via source_ids JSON array.

Example: "Oz has 4 RTX 3090s" + "the GPU server is the GPU server with 4x 3090" → single merged memory.

Merged memories get consolidated links back to each archived source, supplementing the source_ids JSON with graph-queryable edges.

Memory Linking

Memories form a relationship graph. Three link types are created automatically:

Type	Created when	Strength
`co_extracted`	Multiple facts extracted from the same exchange	1.0
`consolidated`	Memories merged during consolidation	1.0
`semantic`	Embedding cosine similarity > 0.85 (discovered during maintenance)	Cosine similarity score

Links are directed in storage but queried bidirectionally. Duplicate edges are prevented by a UNIQUE(source_id, target_id, link_type) constraint.

During retrieval, after hybrid search returns primary memories, the graph is walked 1-hop to pull in up to 3 additional linked memories — enriching context even when the linked facts don't match the query directly.

Links referencing archived memories are cleaned up automatically at the end of each maintenance cycle.