Muninn Features

Source-of-truth feature inventory for Coabai Muninn. Tracks what is shipped in main at the latest tag. Roadmap items live in the issue tracker and per-version project_muninn_* notes — not here.

Current baseline: v0.12.0 shipped + deployed (2026-05-30). Tip-of-main has run well ahead of the last tag (schema v31). On top of v0.12.0 it adds: the music tool, conversation replay capture + muninn replay, the reflective "dream" pass; DeepSeek as the primary brain with a fallback brain + round-robin peer pools; presence / co-location with a family map (people layer + custom pins); open-house voice mode; the email / calendar / contacts / youtube / research tools; subtitles (Bazarr), mastodon, photos, geocode / route maps tools, media_request / music_request (Jellyseerr / DJ request), code_task (delegate coding to Huginn over NATS), scratchpad (per-task working memory), identity / workspace_edit / self_restart; one-time (run_once) schedules; an Alertmanager → Matrix relay with generic auto-remediation; and a failure-learning loop (run-outcome ledger + nightly review → constraint memories).

Legend: [x] shipped · [?] partially shipped or behind a feature flag · [ ] planned (kept out of this file).

Transports

• Matrix — first-class transport. Olm/Megolm E2EE, cross-signing, device verification, sync loop, allowed-user gate.
• TUI — terminal chat surface (muninn-tui launcher). Separate config + DB, peer-reads the daemon's Matrix DB read-only so memories are shared.
• Web — third surface inside the daemon. Axum HTTP, OIDC bearer auth (any issuer with a userinfo endpoint), default bind 127.0.0.1:7792. Also serves the family map JSON endpoints (see Presence & Family Map): /map/people (live GeoJSON people layer) and /map/pins (GET list / POST create / DELETE), both behind the same presence_map_allowed_users allowlist via a shared authorize_map() gate — used by a self-hosted server.
• Voice (M2) — fourth surface. Axum HTTP POST /api/voice/event from voice-hub; resolves pyannote_id → user_id and device_id → room via the M1a/b registries; optional shared-secret auth; reply path TTS-routes via the native tts::TtsRouter (HA tts.speak, [tools.tts]) or the standalone voice-server /speak fallback, using the device's media_player. Default bind 127.0.0.1:7793.
• Embedded docs server — static HTML in docs/ served by axum on :7790 (daemon) / :7791 (TUI). Edit + refresh, no recompile.
• Pluggable transport trait — Transport + IncomingMessage in src/transport/mod.rs; new transports drop in alongside Matrix.
• Session persistence — data/session.json for device_id + access_token; data/matrix-store/ SQLite crypto store. Restores on startup, falls back to fresh login on failure.

Agent Loop

• OpenAI-compatible LLM client (src/llm.rs) — chat_with_tools() for the main loop, send_request_raw() for sub-agents, chat_with_system() for extraction. Primary brain in prod is DeepSeek (muninn.toml.local), Qwen stays a peer/fallback.
• Fallback brain — [llm] fallback_url / fallback_model / fallback_api_key on a separate host: on a transport error or 5xx from the primary, the non-streaming path retries once on the fallback (5s connect-timeout so a dead primary fails over in seconds; read timeout stays unset so long generations aren't cut off). Logs the failover at WARN. Unset = no failover (back-compat).
• Round-robin peer pools — [llm] api_peers / fallback_peers: extra interchangeable endpoints (same model on another host) round-robined alongside the primary so conversation / tool / extraction load spreads across boxes; a dead peer is skipped. Classifier/fast clients drop the peers (different small models, own hosts).
• Per-tool progress + thinking preface — ChatResult::ToolCalls carries a preface; a "→" progress line is emitted per tool call (with the binary basename + subcommand for shell, privacy-safe), so the user sees what's happening before each tool fires.
• Mid-task steering — !stop plus natural-language steering (SteerMsg/SteerRegistry; a classifier routes steer-vs-new-message, fails open). Background tasks get content-derived names (gen_bg_name).
• Single agent router tool in main context (~150 tokens) — saves ~70% on tool definition tokens vs exposing all tools.
• Sub-agent execution — ephemeral context, full tool registry, results returned to the main LLM as the agent tool result.
• Three-round tool loop — both main and sub-agent. Last round strips tool defs to force a text reply.
• Per-request sampling params — temperature, top_p, top_k, presence_penalty, reasoning_budget (config-driven, model-specific).
• Static system prompt — user/location facts baked in to preserve llama.cpp KV cache across the day.
• reasoning_content capture — model thinking logged when the server emits it.
• SSE streaming via futures-util for incremental responses.
• Markdown → HTML rendering via pulldown-cmark for Matrix HTML message bodies.
• Garbage guard — drops replies that are pure tool-syntax leaks, JSON fragments, or identifier dumps.
• Engagement-fishing strip — trims trailing "Want me to…" questions added by the LLM.
• Loop detector — short-circuits runaway tool patterns (same call repeated, etc.).

Tools (agent router + inner tools)

Sub-agent inner tools, registered in main.rs. The agent router is the only tool the main LLM sees; a few benign tools (e.g. geocode, route, photos) are also flattened onto the main LLM so it doesn't have to delegate for them. Each tool is registered only when its config block is present, so the live surface depends on the deployment. Page fetching is no longer a native tool — it moved to the muninn-fetch MCP server (see Misc), so web_fetch was removed.

Information & Web

• weather — Environment Canada via HA REST API (~11 ms); falls back to Open-Meteo when HA is unavailable.
• web_search — SearXNG self-hosted, Tor-proxied (~1–3 s).
• geocode / route — (optional integration — needs [tools] maps_url pointing at an OSM maps stack; absent → not registered) read-only wrappers over a self-hosted OSM maps stack (Nominatim/Photon + Valhalla behind Caddy). geocode does forward (place/address → coords) and reverse (coords → address) lookups via worldwide Photon. route returns distance + time between two places/coords by car/bike/foot (resolves place-name endpoints through the geocoder first), pins units to km, and attaches a traced map snapshot — it decodes Valhalla's leg shapes, decimates to ≤150 points, and renders the line over the Protomaps basemap (tileserver-gl /render), attaching the PNG via the same media queue as camera (best-effort; a render failure still returns the text answer). Tiles are planet-wide; routing covers North & Central America + Europe. Both are always-on (benign, like weather) and flattened onto the main LLM.

Compute & Shell

• calculate — Python expression eval, sandboxed subprocess (env_clear, stdin pipe, 10 s timeout).
• shell — Bash with two-gate safety: hardcoded fast-reject + LLM safety judge.
• code_run — Python code execution (TUI sandbox mode).

Filesystem

• file_read / file_write / file_list — sandboxed FS access.
• project — project file enumeration.

Home Assistant

• home_query — HA /api/conversation/process read side ("is the basement light on?"). Soft regex guard rejects action verbs.
• home_action — HA /api/conversation/process write side ("turn off the basement lights"). Split from legacy home at M1c so M1d's role filter can gate writes independently.
• camera — Frigate snapshot + the GPU server-vision description, parallel fetch of motion / person / face sensors, uploads JPEG to Matrix.
• music — Music Assistant control via HA. Play modes: by name; favorites+random ("play some music"); discover (random library minus favorites — "surprise me"); radio (radio_mode similar-tracks station — "more like this / X radio"); genre (playlist search — "play some jazz / 90s music"). Plus pause/resume/stop/next/previous, volume (set/nudge/mute), shuffle, favorite/dislike (skip + remember), transfer, synced join/unjoin (multi-room), "pause everything", and status (track + volume). Room-aware targeting (config map → HA area → device → fuzzy); grouped-pause doesn't cascade; slow-device playback is backgrounded so the turn never hangs. [tools] music_assistant_config_entry_id enables search/library/genre. Full reference: MUSIC.
• DJ mode (radio host) — (optional — [tools.dj] enabled, off by default; needs Music Assistant + a configured TTS router) the music tool's dj action ("be a DJ for the basement" / "stop") flips radio-host mode on per player. A background worker polls each active player (~3s) and, ~5s into every new track, speaks a short on-air intro over the music — ducked via the TtsRouter announce path, so the song dips under the voice and recovers (no mute/jump). The announcer persona is picked by the song's detected language; the patter vibe rotates each song (round-robin) so it never repeats. Personas + vibes live in an editable markdown file ([tools.dj] md_file, default dj.md), re-read per announcement so you retune without a rebuild (missing/unparseable → baked-in defaults). Optional web grounding (web_grounding, default on) runs a quick SearXNG search so the patter cites real artist/song facts instead of hallucinating (6s budget → fact-free fallback; inert without searxng_url). The active-player set is persisted (JSON) so DJ mode survives a restart; a forgotten session auto-expires after idle_timeout_secs (default 30 min) of non-playback. Config [tools.dj]: enabled, md_file, poll_secs, announce_after_secs, idle_timeout_secs, web_grounding.
• photos — (optional integration — needs [tools.photo_displays] + Home Assistant + an Immich slideshow on a wall display; absent → not registered) show/hide an Immich photo slideshow on an esphome-android wall display. Resolves a place → [tools.photo_displays] prefix → HA switch.<prefix>_show_photos (ON via switch.turn_on, OFF via switch.turn_off). Place defaults to the voice ctx.room; asks if a typed command omits it. Flattened onto the main LLM. admin/adult/child.
• presence — (optional integration — needs [tools] presence_enabled + Home Assistant device_trackers; off by default) "who and what is where". Thin wrapper over the shared PresenceEngine (src/presence.rs): fuses configured users' HA device_trackers, auto-detects cars, and clusters co-located people (named zone, or haversine distance under presence_colocate_meters). Enforces GPS freshness — a fix older than presence_stale_secs reads as "location unknown" rather than a ghost location. Answers "is anyone home?", "where is a family member?", "are Oz and Maria together?". Config [tools] presence_enabled. Allowed for admin/adult/child (read-only household presence is benign).

Media Generation

• imagine — ComfyUI image gen (Flux 2 Klein 9B). Text-to-image and image-to-image. Async with placeholder image, swapped when generation lands. LLM caption held and attached.
• speak — native HA tts.speak via tts::TtsRouter ([tools.tts]): speaker/voice resolution, Kokoro↔OmniVoice engine dispatch, notify.* phone targets, strip_noise+fix_pronunciation; standalone voice-server fallback when unconfigured.
• transcribe — STT. Default path is the HA STT REST API; when [tools] stt_model is set it POSTs audio straight to an OpenAI-compatible /v1/audio/transcriptions endpoint (speaches/Parakeet on the GPU server), skipping the HA + Wyoming hops and the entity-name fragility (multipart file+model+language; omits the auth header when the token is empty so the HA bearer never leaks).

Email

• email — IMAP read/search/flags + SMTP send over one or more mailboxes ([tools.email], default gateway a self-hosted server / Mailcow a self-hosted server). Actions: accounts, list (folder, unread-only), read (by uid, non-destructive PEEK), search (IMAP TEXT), send (to/cc/bcc), reply (threaded via In-Reply-To/References, optional reply-all), mark_read/mark_unread, archive, delete (Trash or permanent), folders, extract_events. Multi-account: one [[tools.email.accounts]] block per mailbox; calls pick the default (or named) account. Sent copies are IMAP-appended (best-effort). imap/lettre are blocking → run in spawn_blocking. Admin-only at the role gate; master enabled toggle gates registration. Reference: docs/references/odysseus/.
• email → extract_events — appointment mining, confirm-first: scans one uid or the recent INBOX (unread_only/limit), runs a per-email LLM pass (date-grounded to Atlantic, mail treated as untrusted) → [{title, when, all_day, location, notes, confidence}], and returns a numbered proposal list. Never writes the calendar itself — the agent confirms with the user, then calls calendar add per pick. Dedup (v28): each processed message is recorded in email_calendar_extractions (Message-ID keyed) so a re-scan skips it — no re-proposing, no wasted LLM pass; rescan=true overrides. Prompt adapted from Odysseus email_pollers.py.

Calendar

• calendar — appointments + timed reminders. add (NL when like "next Tuesday at 3pm" → resolved against Atlantic time via the utility LLM; optional location/notes/duration/reminders), list, remove. Local SQLite is source of truth (calendar_events + calendar_reminders, schema v25); best-effort mirrored to SOGo CalDAV (a self-hosted server) so events show in real calendar clients. Default reminders 1 day + 1 hour before (configurable, per-event overridable); the calendar reminder worker (src/tools/calendar.rs, 60s tick) posts each due reminder to the originating Matrix thread via the same send_notice path as the scheduler. Multi-account via [[tools.calendar.accounts]]. One shared calendar serves all users: every event records who added it (created_by) and a public/private visibility (v26) — list shows public events to everyone plus the caller's own private ones (🔒). Recurring events (v27): NL → iCal RRULE (rrule crate, DST-safe), pushed to CalDAV as a series; the worker rolls a 60-day horizon each tick to materialize per-occurrence reminders; remove cancels the whole series. Admin-only.

Contacts

• contacts — CardDAV name→email/phone lookup + add over the SOGo Personal Address Book ([tools.contacts]). Actions: search (match name/email/org — resolves "email a contact" to an address), list, add (name + email/phone). One addressbook-query REPORT pulls every vCard inline, parsed by a hand-rolled vCard reader (FN/EMAIL/TEL/ORG, structured-N fallback, mailto: stripping) — no CardDAV/vCard crate. add PUTs a vCard 3.0. Admin-only. The agent resolves a name here, then calls email send — no cross-tool coupling.

Media

• youtube — summarize a YouTube video from its transcript ([tools.youtube]). yt-dlp (host binary, argv — no shell) pulls auto-captions as VTT; a hand-rolled VTT cleaner strips timing/tags and collapses the rolling-duplicate caption lines → transcript → LLM summary (TL;DR + key points + notable moments). URL is host-validated to YouTube (no open-fetch SSRF). 90s subprocess budget; graceful errors for no-captions / not-found / timeout. admin/adult.
• subtitles — (optional integration — needs [tools.bazarr] + a running Bazarr; [tools.jellyfin] further optional; absent → not registered) conversational front-end over Bazarr for "get subs for <title>". Resolves a movie (/api/movies + /api/providers/movies) or TV show (/api/series + /api/episodes + /api/providers/episodes) in Bazarr, runs a live provider search, and downloads the best match per wanted language. It re-checks what actually landed rather than trusting the POST (Bazarr discards below-threshold/failed-sync subs even after a 204). TV args season/episode → one episode / a whole season / all missing episodes (bounded inline). A whole-season/series ask bigger than the inline cap runs in a background worker when a Matrix room is available (mirrors code_task): the tool replies "working on it" and posts a summary back to the room when every episode is done; voice/scheduler contexts keep the bounded inline behavior. After a successful download it triggers a Jellyfin library scan (the only thing that re-discovers external subs; incremental, ~12s). admin/adult/child.
• media_request — (optional integration — needs a Jellyseerr config block + a running Jellyseerr; absent → not registered) request a movie/show via Jellyseerr (gates "on Jellyfin ✅" only on a real jellyfinMediaId). Kids can ask; child requests are held for owner approval. admin/adult/child.
• music_request — (optional integration — rides on the music stack, so it needs Music Assistant installed in Home Assistant) request-side companion to the music/DJ feature: kids can ask for music, owner-gated. admin/adult/child.
• mastodon — (optional integration — needs [tools.mastodon] + a Mastodon account/instance; absent → not registered) native Mastodon tool, replacing the retired OpenClaw cron for the daily "On This Day in History" image posts. Two actions: recent (today's Atlantic date + the account's last N posts, for self-contained dedup of which event to post) and post (generate an image via the shared imagine/Ideogram pipeline or take a local path, upload it as media, then post the status). Routed to the QUALITY tier (multi-step compose). Admin-only. Driven 3×/day by the scheduler.

Research

• research — deep research ([tools.research]): its own bounded plan→search→fetch→synthesize loop → cited markdown brief. A grounded planner (one preliminary SearXNG search informs the decomposition) splits the question; each sub-question hits SearXNG; top unique URLs are fetched concurrently (SSRF-guarded — public http(s) only) and reduced to text. Embeddings context compression (ports GPT-Researcher's EmbeddingsFilter): each page is chunked + embedded via the [memory] Ollama embedder, keeping only the chunks most similar to the query within max_page_chars — synthesis sees the relevant slice, not the first N chars (fails open to truncation without embeddings). Iterative deepening (deepen_rounds, default 1): after the first gather, the LLM reflects on coverage gaps and runs targeted follow-up search rounds (raises the source ceiling 10→14; fails closed). Synthesized into an in-depth, multi-section report (synthesis runs at synthesis_max_tokens, default 6000, so it isn't clipped) with an auto-appended ## Sources list. Bounded (≤10/14 sources, 40k synth chars, 8s/fetch); slow → run under !bg. Direct-return (not re-summarized). admin/adult.

Memory & Meta

• memory_lookup — direct human-readable memory search.
• progress — tool execution progress reporting.
• scratchpad — per-task working memory (set/append/get/list/clear) — fixes sub-agent statelessness across rounds. Backed by task_memory (runtime_kv), scoped to (channel_id, topic_id), 3h TTL, bounded, fails open, unknown senders excluded. Auto-carried into every sub-agent run as a ## Working memory block.
• schedule — cron schedule create / list / delete, plus one-time (run_once) schedules (v31): a NL time like "remind me at 3pm tomorrow" resolves to an absolute UTC instant (Atlantic-grounded via nl_to_oneshot), pins the cron to that minute, and the row is archived after its single fire (so a mid-run crash retries).
• agent — router that delegates to a sub-agent (the only tool in main context).

Self-Modification & Delegation

• code_task — delegate a coding job to Huginn over NATS, instead of writing runnable code inline. create_workspace=true makes Huginn create a brand-new named workspace for a fresh artifact (script / small app / canvas demo); false (default) targets an existing repo (Huginn refuses an unknown name, catching a typo'd repo). Live progress relay, structured outcomes, and mid-flight steering (code_task + a steer tool). When it hands off, Muninn publishes a SpawnChild envelope on the Hermes feed bus (coabai-agent-protocol) keyed to the job id, so the Hermes console opens an interactive Huginn child tab that fills with the job's real progress (best-effort; dropped when Hermes isn't listening). Admin-only.
• identity / workspace_edit / self_restart — self-modification tools (persona overlay; edit own config/persona workspace files; bounce own service with a detached safe restart + loop-guard). Admin-only.

Memory System

Persistent SQLite-backed memory at data/muninn.db, FTS5 + sqlite-vec, schema v31.

Storage & Search

• Append-only messages raw log.
• Mutable memories table with importance, strength, access_count, source_ids, archived_at, user_id, scope, corrects.
• vec_memories virtual table — 768-dim embeddings (Ollama nomic-embed-text).
• memory_links graph: co_extracted, consolidated, semantic link types with strength.
• FTS5 messages_fts + memories_fts kept in sync via SQL triggers.
• Per-channel maintenance_state for incremental workers.
• conversation_summaries cache — one rolling summary per channel, keyed by last_message_id.
• vec_skills table — embeddings on extracted skills.
• Hybrid retrieval — FTS5 + vector RRF, 1-hop graph augmentation, strength-weighted BM25 (strength × (1 + importance)).

Lifecycle Workers

• Async fact extraction — every exchange spawns a tokio task; LLM extracts facts, dedups via FTS5, upserts, embeds. Does not memorize network/host facts — the extraction prompt forbids IP/port/VLAN/host-spec/scan/status facts (an explicit "remember <net fact>" still wins), and a deterministic backstop (is_transient_infra, mentions_ip_or_vlan) drops scan/status/inventory forms and auto-extracted IP/VLAN facts so heavy SSH/scan sessions stop leaking transient tool output into never-decaying "facts". Network inventory lives in the workspace network-map file instead.
• Correction detection — corrects field; substring-matched contradicted memories are auto-archived before the new fact is inserted.
• Maintenance worker (30-min timer):
  • Episodic summaries (groups of 6+ messages within 60-min windows → LLM summary).
  • Fact consolidation (Jaccard-similar memories → LLM merge, originals archived with source_ids provenance).
  • Ebbinghaus decay (strength × 2^(-days/14), archived below 0.1; access resets the clock).
  • Semantic linking (cosine > 0.85 creates links).
  • Embedding backfill.
  • Link cleanup.
  • Skill curator (v0.12) — archives skills that are unused since creation, low-success after enough uses, or idle past archive_idle_after_days; toggles + thresholds under [memory] skill_curator_*.
  • Per-user dialectic profile (v0.12) — distills each user's top-importance personal memories into a hard-capped (profile_max_chars, default 1500) "About <user>" block; refreshes on profile_refresh_after_hours; injected into the system prompt on every turn from that sender.
• [?] Reflective "dream" pass (v24, schema maintenance_state.last_dream) — opt-in nightly consolidation that re-reads recent messages and distils higher-order memories/links beyond the per-cycle extraction. Config: [memory] dream_enabled (default false), dream_interval_hours (24), dream_max_messages (120).
• Rolling conversation summaries — older messages aged out of recent window are summarised, cached, refreshed asynchronously when stale, prepended to context as [Earlier in this conversation].
• Skill extraction worker — distills successful sub-agent trajectories into procedural memory; toggle: [memory] skill_extraction_enabled.
• Skill curator (v0.12) — periodic janitor for the skills table (see Maintenance worker above). Manual run: !curate-skills. Inspection: !archived-skills.
• User profile worker (v0.12, Honcho-lite static phase) — generates and refreshes the user_profiles table (schema v22). Manual ops: !profile [user_id], !regenerate-profile [user_id].
• Query-aware profile slicer (v0.12, full Honcho) — when [memory] profile_slicer_enabled = true, every turn from a known user makes one extra LLM call against classifier_llm (the GPU server-small in prod) to extract the slice of the cached profile most relevant to the incoming message. Three outcomes: relevant slice → inject sliced; nothing relevant → omit profile this turn; LLM failure → fall back to full profile. Default OFF — opt in after observing baseline latency from B1.
• Failure-learning loop (v30/v31) — the failure mirror of the skill loop (which only captured successes). Every sub-agent run records a RunOutcome (ok / ok-with-failures / round-limit / empty / error) plus its signals (steps, guard nudges, tool failure, round-limit hit, tools used) to the run_outcomes ledger; capture is best-effort and never recorded for unknown speakers. A run is also marked corrected when the user corrects Muninn in the next exchange (hooked into the existing correction detection) — a corrected "ok" is a quiet failure, excluded from the clean-rate. failures.rs analyzes the ledger (outcome distribution, clean rate, round-limit / tool-failure rates, unhealthy-tool hotspots) → !failures. Nightly review → constraints: !review-failures reads recent unhealthy/corrected runs, distils up to 3 recurring anti-patterns into short constraint memories that surface back through normal retrieval (failure → constraint → surfaced → avoided); opt-in auto via [memory] failure_review_enabled (default OFF, throttled to ~once/24h).

Scoping

• Per-user isolation — personal / room / global tiers via user_id + scope.
• Per-topic scoping — see Topics & Threads.

Identity Registry

Canonical user_id bridging every surface a person speaks through into one row. M1a of the wake-word-free voice work — see project_wake_word_free_voice memory.

• users table (schema v19) — user_id PK, display_name, matrix_mxid, pyannote_id, tts_voice, role, allowed_rooms (JSON), source, timestamps. Unique partial indexes on matrix_mxid and pyannote_id.
• Config seed — [[users]] blocks in muninn.toml upsert at every boot, marked source='config'. Configured fields always win on conflict.
• UserRegistry (src/identity.rs) — by_id, by_matrix_mxid, by_pyannote_id, list, insert_runtime. Holds its own SQLite connection on the same data/muninn.db.
• Roles — admin | adult | child | guest | self.
• [?] Lookups not yet wired — registry held by BotCore, but resolver calls land in Phase M2 (voice transport) and M1d (role-filtered ToolRegistry).

Device Registry

Stable per-surface ids that voice events arrive under, mapping to room and HA media_player. M1b of the wake-word-free voice work.

• [[devices]] config seed — in-memory DeviceRegistry (src/devices.rs) loaded from muninn.toml at boot. No DB persistence (devices are infrastructure).
• Resolvers — by_id, in_room, list. Synchronous.
• media_player polymorphism — accepts voice-server alias (the workstation, kitchen, pixel 6a) or raw media_player.* entity.
• room = "varies" — supported for mobile devices (phones); voice transport will need a per-utterance room hint.
• [?] Lookups not yet wired — registry held by BotCore, callers land in Phase M2 (voice transport).

Presence & Family Map

Optional — off until [tools] presence_enabled is set; the map layers further assume Home Assistant device_trackers and a self-hosted OSM map front-end (e.g. a self-hosted server). A shared PresenceEngine (src/presence.rs) plus a custom-pin store, surfaced both as the presence tool (above) and as JSON layers for the map.

• Co-location engine — fuses configured users' HA device_trackers, auto-detects cars, and clusters co-located people by named zone or haversine distance (presence_colocate_meters). GPS freshness enforced: a fix older than presence_stale_secs reads as "location unknown" rather than a ghost location.
• People layer — GET /map/people returns bearer + allowlist-gated GeoJSON (Markers). When a stationary phone stops reporting coords (last fix aged out) but HA still places it in a known zone, the marker falls back to the zone center (approx: true) so family members stay visible; only genuinely-unknown (away, no fresh fix) is omitted.
• Custom pins — PinStore (own SQLite table in data/muninn.db, mirrors UserRegistry): drop / label / delete places on the family map. GET /map/pins (list) · POST /map/pins (create) · DELETE /map/pins/{id}. created_by is the authenticated user (never the request body); label + lat/lon validated. Same presence_map_allowed_users allowlist as the people layer via the shared authorize_map() helper.
• Config — [tools] presence_enabled, presence_stale_secs, presence_colocate_meters, presence_map_allowed_users.

Open-House Voice Mode

• [tools] open_house (default false) — when on, every speaker (recognized, guest, or unknown) may use the house-control surface: home_query / home_action, music, photos, presence, speak, and media_request / music_request / subtitles. Family-private surfaces stay gated even in open mode — camera (surveillance) and memory_lookup (personal data) keep their normal role gate, as do all admin/dev tools. Goal: low-friction adoption + usage/clip data collection. Enforced in tools::agent::role_allows via a process atomic set at boot (set_open_house); reversible with open_house=false + restart.

Role-Based Tool Filter

M1d. The sub-agent's tool surface is filtered per request by the caller's role.

• role_allows(role, tool_name) — pure function in src/tools/agent.rs encoding the permissions matrix (admin / adult / child / guest / unknown / self).
• AgentTool::resolve_role — looks ctx.sender up in UserRegistry (Matrix mxid first, then canonical user_id); misses → "unknown".
• Visibility filter — all_tool_defs(ha_access, role) excludes denied tools from the sub-agent's system prompt and tools array.
• Execute-time gate — execute_tool() rechecks role_allows() and returns "Access denied: role 'X' may not invoke Y." as defense in depth.
• AND with ha_allowed_users — the older sender-explicit allowlist for HA tools is layered with the role gate. A user must pass both.
• MCP gating — admin-only for now; per-server promotion case-by-case.
• [?] Finer-grained gating (room-limited home_action, own-voice-only speak, own-scope memory_lookup) — allowed at the AgentTool layer; tightening lands inside each tool with the M2 voice transport.

Ephemeral Sessions (Unknown Speakers)

M3a. RAM-only conversation state for sender == "unknown" voice events. Bypasses the memory DB entirely.

• EphemeralSessionStore (src/sessions.rs) — keyed by device_id, TTL 5 min idle, bounded turn buffer (16 turns / 8 pairs). Background purge worker on TTL/2 tick.
• BotCore::handle_unknown_message — dedicated path that short-circuits BEFORE topic resolution, memory log, and memory extraction. Single-shot LLM call with the agent router; tool surface restricted by the M1d role gate to weather + calculate + transcribe + progress.
• Privacy-guard system prompt — EPHEMERAL_SYSTEM_PROMPT replaces the workspace persona for unknown turns. Decline household questions; suggest contacting a resident for state-changing requests.
• Lost on Muninn restart by design.

Onboarding (M3b)

Deterministic state machine on each ephemeral session. Runs as a parallel track — the conversation isn't blocked on identification.

• States — NotStarted → AskedName → NameCaptured(name) → ClaimedKnownName | AwaitingApproval.
• Triggers — Strategy C (identity-class regex on user text) + Strategy B (8-turn chit-chat safety net). Weather / time / public info don't trigger.
• Name extraction — regex patterns for I'm X, my name is X, this is X, X here, plus bare-word reply. is_plausible_name rejects greetings/fillers.
• Impersonation defense — claimed name matched against display_name in UserRegistry (case-insensitive by_display_name_ci). On match, system-prompt addendum tells the LLM to acknowledge warmly but never confirm.
• [?] Cross-channel approval (DM the owner, 5-min timeout, pyannote /enroll) — stubbed in v0; the audit table from M3c is the foundation for the DM correlator.

Auth Events Audit Log (M3c)

Append-only audit log for unknown encounters and onboarding state transitions. Schema v20.

• auth_events table — id, ts, device_id, event_type, pyannote_id, pyannote_confidence, claimed_name, matched_user_id, transcript_preview (capped at 200 chars).
• AuthEventStore (src/identity.rs) — async writer holding its own SQLite connection; failures warn-logged and swallowed (never deny service for an audit hiccup).
• Event vocabulary — unknown_encounter, name_asked, name_captured, claimed_known, enrollment_requested, plus reserved enrollment_approved / enrollment_denied for the future approval flow.
• Wired — BotCore::handle_unknown_message writes one row per unknown turn + one per state transition.

Topics & Threads

• LLM topic classifier — picks an existing topic or proposes a new one per inbound message.
• Topic-scoped memory retrieval.
• Topic lifecycle — dormant aging.
• User-initiated threads only (schema v18) — classifier tags but never spawns Matrix threads.
• General-topic posts go to room root, not threaded.
• Commands: !topics, !topic, !archive, !retag.

Vision & Media Pipeline

• Frigate snapshot fetch via HA.
• Vision describe via the GPU server LLM (same llama.cpp instance).
• Parallel Frigate sensor fetch (motion, person count, face recognition).
• Image-to-image style transfer from camera snapshots.
• SharedMediaQueue — tool → bot image hand-off.
• Background job queue for async generation with placeholder image swap.
• Matrix JPEG / PNG upload.

Home Assistant Integration

• Weather: GET /api/states/weather.* + POST /api/services/weather/get_forecasts?return_response.
• Conversation API: POST /api/conversation/process.
• HA-side entity resolution — Muninn passes natural language, no entity-ID knowledge required.
• Long-lived access token via [tools] ha_token.

Scheduling

• schedules table with cron + room + thread + task fields (+ run_once, v31).
• Tick worker — wakes on the minute, fires due cron tasks via sub-agent. Claim-first double-fire fix: a schedule's next_run_at is advanced before the task runs (claim_schedule), so a multi-minute task isn't re-fired by later 60s ticks.
• One-time (run_once) schedules — fire once at an absolute time, then retire. The row is archived after the fire (not at claim) so a mid-run crash retries.
• 5-field cron expressions (cron crate).
• Per-schedule room and thread delivery.
• Heartbeat worker — reads HEARTBEAT.md, evaluates pending tasks, posts ack. Addresses the user by [heartbeat] address_as (name woven into the eval prompt + reminder greeting; name-neutral when unset).
• schedule tool for LLM-driven schedule management.

Observability & Export

• ⏱ timing instrumentation — memory context, LLM calls, tool exec, sub-agent rounds, end-to-end. Visible in journalctl --user -u coabai-muninn.
• Context logging — data/logs/YYYY-MM-DD_<room>.log, async writes, 1 K line cap. Internal extraction/maintenance calls excluded. Tool args/results redacted unless [llm] log_tool_results = true.
• Replay capture ([llm] replay_capture, default on) — every conversation LLM exchange written as a replayable JSONL record under data/logs/replay/, one file per session (delimited by !clear), last 5 sessions per channel retained. Each record holds the exact built request (system prompt, messages, tool defs, sampling params) + the raw response — POSTable straight back to the LLM.
• muninn replay CLI — re-run a captured record against the LLM, optionally swapping the system prompt (--system-file) to A/B-test prompt changes; --list / --line N select the record.
• muninn export-trajectories CLI — Atropos-compatible JSONL, runs alongside the daemon (read-only).
• Export filters: --since, --channel, --topic, --min-tool-calls, --include-system.
• Status reactions on Matrix messages: 🤖 agent, 🔍 search, 🌐 fetch, 💻 shell, 🧮 calc, 📷 camera, 🎨 imagine.
• Credential scrubbing — regex redaction of tool output before LLM and before persist.

Infra Alerts & Auto-Remediation

Optional — off until an [alertmanager] block is configured; assumes an external Prometheus/Alertmanager pointed at the webhook, and each remediation block names its own command.

• Alertmanager → Matrix relay (src/alertmanager.rs) — axum webhook receiver that turns firing Prometheus/Alertmanager alerts into Matrix messages in an ops room; infra alert text names the failed service/pod.
• Generic auto-remediation — a firing alert whose alertname matches an [[alertmanager.remediation]] block runs that block's command (argv, no shell, 20s timeout, per-alertname cooldown) and appends the outcome to the Matrix alert. RemediationConfig { alertname, command, cooldown_secs, description }. Reusable for any self-heal — add a config block, no code change.
• First use: Onkyo AVR WiFi self-heal — the OnkyoUnreachable alert (label remediation: onkyo-deauth) SSHes to AP .6 via a key locked to a single forced-deauth command to kick the receiver into re-associating + pulling a fresh DHCP lease (~4s recovery); if that fails, the 12m critical rule escalates to a phone push.

Deployment & Configuration

• systemd user unit (coabai-muninn.service.example → ~/.config/systemd/user/).
• ./deploy.sh rebuild + restart.
• TOML config with MUNINN_CONFIG env var precedence (falls back to muninn.toml).
• Sections: [matrix], [llm], [memory], [tools], [tools.tts], [tools.email], [tools.calendar], [tools.contacts], [tools.youtube], [tools.research], [tools.bazarr], [tools.jellyfin], [tools.mastodon], [tools.dj], [tools.music_rooms], [tools.photo_displays], [docs], [tui], [web], [voice], [heartbeat], [audio_companion], [alertmanager], and array tables [[users]], [[devices]], [[persona_voices]], [[tools.mcp_servers]], [[tools.email.accounts]], [[tools.calendar.accounts]], [[alertmanager.remediation]].
• data/ and muninn.toml.local gitignored.
• Two-step UIA cross-signing bootstrap in setup_e2ee().
• Shared peer-DB pattern for the TUI surface (read-only against daemon DB).

Commands (Matrix / TUI)

Misc

• MCP client in src/mcp.rs — both stdio JSON-RPC and HTTP / streamable-http (FastMCP) transports ([[tools.mcp_servers]] with kind = "stdio" | "http"). MCP tools are admin-gated. muninn-fetch (HTTP MCP) supplies page-fetch/screenshot/extract, replacing the removed native web_fetch.
• Voice persona stack — [[persona_voices]] map a transcript opener (e.g. a character name) to a TTS voice + a per-turn system-prompt addendum; persona sticks per-device for [voice] persona_ttl_seconds. Topics are persona-scoped (schema v21) so personas don't share conversation history.
• Audio companion — [audio_companion]: when an inbound Matrix message was a voice note, send a TTS audio reply alongside text; long replies summarised first.
• Atomic message batching per topic.
• Standalone OIDC test page (web-test/) for the local auth server.

Versioned Highlights

Roadmap (not in this file)

Tracked in the project_muninn_* notes outside the repo and the issue tracker. Items currently being scoped:

• Voice satellite (LVA → Matrix)
• Speaker ID via pyannote ECAPA + sqlite-vec
• Speaker-gated wake-word-free voice
• Family-scoped multi-bot deployment
• Conversation-quality and proactive-HA polish

Update this file when a roadmap item lands and a tag ships.