Architecture

Sub-agent design, pluggable transports, async background workers

System Overview

Transport(s) ──[IncomingMessage]──► mpsc channel ──► BotCore
                                                    │
BotCore ──[Memory]──► log_and_build_context() ──► LLM (sees only "agent" tool)
                                                    │
                                          [if tool call] ──► Sub-Agent
                                                    │         (ephemeral context)
                                                    │         ├── weather (HA)
                                                    │         ├── home_query / home_action (HA)
                                                    │         ├── music (Music Assistant via HA)
                                                    │         ├── camera (HA + vision LLM)
                                                    │         ├── imagine (ComfyUI)
                                                    │         ├── calculate (Python)
                                                    │         ├── web_search (SearXNG) + muninn-fetch (MCP)
                                                    │         ├── speak / transcribe / memory_lookup / schedule
                                                    │         ├── email (IMAP+SMTP) / calendar (CalDAV + reminders)
                                                    │         └── shell (2-gate safety) + file/code (sandbox)
                                                    │
                                          log_response() ──► Transport
                                                    │
                                          [async] extraction worker ──► memories DB
                                          [async] maintenance worker (30min timer)
                                          [async] schedule tick worker (60s) ──► cron tasks
                                          [async] calendar reminder worker (60s) ──► appointment reminders
                                                    ├── episodic summaries
                                                    ├── fact consolidation + correction detection
                                                    ├── Ebbinghaus decay
                                                    ├── embedding backfill
                                                    ├── semantic link discovery
                                                    └── link cleanup

Sub-Agent Design

The main LLM sees only one tool definition (agent, ~150 tokens), keeping the conversation context lean. When the LLM decides it needs a tool, it delegates to a sub-agent that runs in an ephemeral context with all available tools.

70% token savings on tool definitions per request
Scales infinitely — adding tools doesn't bloat the main context
Isolates tool execution — sub-agent context is thrown away after each call

The sub-agent has its own system prompt and tool loop (up to 25 rounds; the main loop caps at 8), with a forced text reply on the last round. Before a state-changing task is declared done, an L2 completion check verifies the outcome actually happened and re-asks on a gap. It fires only when a tool mutated state (a file write/edit, code run, or a shell command that actually writes/installs/restarts/deletes) — pure reads like journalctl or cat skip it, so "show me X" requests don't loop.

Source Modules

File	Role
`main.rs`	Startup: config, LLM, memory, inner tools, agent wrapper, transports, docs server
`bot.rs`	Core logic: command dispatch, LLM routing, tool loop, timing instrumentation, emoji reactions
`llm.rs`	OpenAI-compatible HTTP client (chat, tool calls, per-request sampling params, context logging)
`config.rs`	TOML config structs with optional fields and defaults
`embedding.rs`	Ollama HTTP client for nomic-embed-text (768-dim)
`media.rs`	Shared queues: PendingImage (sync), BackgroundImageJob (async generation)
`mcp.rs`	MCP client — stdio JSON-RPC + HTTP/streamable-http (FastMCP). In production for `muninn-fetch` (page fetch/screenshot/extract); admin-gated
`transport/mod.rs`	Transport trait + IncomingMessage struct (text + optional image_data)
`transport/matrix.rs`	Matrix E2EE implementation: sync loop, session persistence, image download

Tool Modules

File	Role	Response time
`tools/mod.rs`	Tool trait + ToolRegistry	—
`tools/agent.rs`	Sub-agent: wraps all inner tools, runs ephemeral LLM calls	2-6s total
`tools/weather.rs`	HA REST API + Environment Canada forecast	~11ms
`tools/home.rs`	HA conversation API — natural language home control	~100-300ms
`tools/camera.rs`	HA camera snapshot + the GPU server vision LLM + Frigate sensors. Uploads to Matrix	~10s
`tools/imagine.rs`	ComfyUI Flux Klein 9B — txt2img + img2img with async placeholder flow	~40-90s
`tools/calculate.rs`	Python expression evaluator (sandboxed subprocess)	~100ms
`tools/web_search.rs`	SearXNG web search (page fetch via muninn-fetch MCP)	~1-3s
`tools/music.rs`	Music Assistant control via HA (play/queue/volume/transfer)	~100-300ms
`tools/shell.rs`	Safety-judged shell execution (2-gate: blocklist + LLM judge)	~100ms
`tools/email.rs`	IMAP read/search/flags + SMTP send over configured mailboxes (imap/mail-parser/lettre, spawn_blocking)	~0.3-2s
`tools/calendar.rs`	Appointments + recurring (rrule) + reminder worker; local SQLite truth + best-effort SOGo CalDAV push	~0.3-1s
`tools/contacts.rs`	CardDAV name→email/phone lookup + add over SOGo (addressbook-query REPORT, hand-rolled vCard parse)	~0.3-1s
`tools/youtube.rs`	YouTube transcript→summary via yt-dlp (auto-caption VTT → cleaned transcript → LLM)	~5-15s
`tools/research.rs`	Deep research: plan→SearXNG→fetch (SSRF-guarded)→synthesize a cited brief; bounded, background via !bg	~30-90s

Memory Modules

File	Role
`memory/mod.rs`	Public API: Memory struct, context building, stats
`memory/store.rs`	SQLite operations: messages, memories, FTS5, vectors, archive
`memory/retrieval.rs`	Context assembly: recent messages + hybrid FTS5/vector search (UTC → Atlantic time)
`memory/extraction.rs`	Background worker: LLM fact extraction, dedup, correction detection, embedding
`memory/maintenance.rs`	Background worker: episodes, consolidation, decay, backfill, semantic linking
`memory/schema.rs`	DDL constants, versioned migrations (v1 → v28)

Message Flow

Message arrives from Matrix transport via mpsc channel (text or image)

1.5

If image attached: VISION llm.describe_image() — base64 data URI → multimodal LLM call → text description injected into message body. Shows 👁️ reaction during processing.

SYNC memory.log_and_build_context() — INSERT user message, hybrid search (FTS5 + vector RRF) + 1-hop graph augmentation → ContextBundle (~40-80ms)

llm.chat_with_tools() — system prompt + memories + recent messages + agent tool def. LLM decides: direct reply or delegate to agent (~1.5-3s)

If agent called → sub-agent runs with all inner tools (typically 1-3 rounds, ~2-6s; up to 25). Tool results flow back to main LLM for final response.

SYNC memory.log_response() — INSERT assistant message, enqueue extraction job

Send reply to transport → user sees response

ASYNC Extraction worker: LLM extracts facts → dedup via Jaccard → upsert memories → correction detection → generate embeddings → create co-extraction links

Context Optimization

Static system prompt — user context baked in, preserves llama.cpp KV cache
Single tool definition — ~150 tokens vs ~500+ for all tools
Timestamps on messages — year included for date awareness without changing system prompt
Memories only when relevant — FTS5/vector search, not always injected
Context logging — data/logs/YYYY-MM-DD.log with timing for optimization

Observability

Every step is timed and logged with ⏱ prefix in journalctl:

⏱ Memory context: 42ms
⏱ LLM tool decision: 1.7s
⏱ Agent initial LLM: 1.9s
⏱ Agent tool home exec: 132ms (20 chars)
⏱ Agent LLM continuation (round 1): 400ms
⏱ Tool agent exec: 2.4s (62 chars)
⏱ LLM continuation (round 1): 500ms
⏱ Total end-to-end: 6.1s

Context logs include full conversation dumps, thinking content (when present), and token estimates.