Tool System
Sub-agent architecture with Home Assistant integration
How It Works
Muninn uses a sub-agent architecture for tool execution. The main LLM sees only
one tool definition (agent), keeping the conversation context lean. When the LLM needs
external data or actions, it delegates to a sub-agent that has access to all available tools.
Two-Layer Tool Flow
agent tool definition (~150 tokens)agent({"task": "Get weather for Moncton today"})Why Sub-Agent?
- 70% token savings — ~150 tokens for one tool def vs ~500+ for six
- Scales infinitely — adding tools doesn't grow the main context
- Clean separation — sub-agent context is ephemeral, thrown away after each call
- The main loop caps at 8 rounds, the sub-agent at 25. The last round strips tool definitions to force a text reply. A loop-detector trip with rounds left now injects a "change tack" nudge and keeps the tools (course-correct, not kill); a garbled/empty final reply is re-asked up to 4× before any fallback.
Tool Trait
#[async_trait]
pub trait Tool: Send + Sync {
fn name(&self) -> &str;
fn definition(&self) -> ToolDefinition;
async fn execute(&self, arguments: &str) -> Result<String>;
}Inner tools are registered in main.rs and passed to the AgentTool,
which wraps them for the sub-agent. The main LLM's ToolRegistry only contains the agent.
Built-in Tools
weather
Current conditions and daily forecast from Environment Canada via Home Assistant.
| Property | Value |
|---|---|
| Backend | HA REST API (/api/states/weather.moncton_forecast + /api/services/weather/get_forecasts) |
| Parameters | {"days": 3} (1-7, default 3) |
| Response time | ~11ms (local network) |
| Output | Current temp/conditions/wind/humidity + daily forecast with highs/lows/precip |
home_query / home_action
Smart home via Home Assistant's conversation API. Split into a read tool (home_query:
"is the basement light on?") and a write tool (home_action: "turn off the lights") so
the role filter can gate writes independently. Natural language in, action out.
| Property | Value |
|---|---|
| Backend | HA /api/conversation/process |
| Parameters | {"command": "turn off the kitchen lights"} |
| Response time | ~100-300ms |
| Capabilities | Lights, locks, sensors, climate, media, covers — anything HA exposes |
HA's intent system handles entity resolution — Muninn doesn't need to know entity IDs.
Example
User: "turn off the basement lights"
Main LLM → agent({"task": "Turn off the basement lights"})
Sub-agent → home({"command": "turn off the basement lights"})
HA → "Turned off the lights"
Reply: "Done, Oz. The basement lights are now off."calculate
Python expression evaluator for math, unit conversions, date calculations.
| Property | Value |
|---|---|
| Backend | Python 3 subprocess via stdin (sandboxed with env_clear) |
| Parameters | {"expression": "2**10 + 3.14"} |
| Timeout | 10 seconds (configurable) |
| Imports | math, decimal, fractions, statistics, datetime, itertools |
| Output cap | 4,000 characters |
web_search
Web search via self-hosted SearXNG routed through Tor proxies.
| Property | Value |
|---|---|
| Backend | SearXNG on .46:8080 |
| Parameters | {"query": "search terms"} |
| Max results | Configurable (default: 5) |
| Output | Numbered list: title, URL, snippet (200 chars) |
music
Play and control music/audio via Music Assistant (through Home Assistant). Use this — not
home_action — for anything involving playing, queuing, pausing, skipping, favoriting,
or volume of music.
| Property | Value |
|---|---|
| Backend | HA + Music Assistant integration |
| Parameters | {"action": "play", "query": "...", "mode": "favorites|discover|radio|genre", "player": "kitchen", "with": "gym", "volume": 0.4, "direction": "up", "enqueue": "play|add|next"} |
| Actions | play, pause/resume/stop/next/previous, volume (set/nudge/mute), shuffle, favorite, dislike (skip + remember), transfer, join/unjoin (synced multi-room), status (track + volume). "pause everything" stops all players. |
| Play modes | favorites (default "play some music" = favorites+random), discover (random library minus favorites — "surprise me"), radio (similar-tracks station — "more like this / X radio"), genre (playlist search — "play some jazz / 90s music"), or a specific named song/artist/album/playlist |
| Player default | Room word resolves via config map → HA area → device registry → fuzzy name; else the speaker's own room, else whatever's currently playing. music_default_player as final fallback. |
| Config | music_assistant_config_entry_id enables search/library/genre; [tools.music_rooms] pins per-room defaults |
| Reference | Full docs: docs/MUSIC.md |
Page fetch (via MCP)
The native web_fetch tool was removed. Page fetch / screenshot / structured-extract
now come from the muninn-fetch HTTP MCP server (configured under
[[tools.mcp_servers]], admin-gated). Use web_search to find URLs, then the
MCP fetch tools to retrieve their content.
camera
Snapshot from any Frigate camera via Home Assistant, with AI vision description.
| Property | Value |
|---|---|
| Backend | HA camera proxy (/api/camera_proxy/camera.<name>) + the GPU server vision LLM |
| Parameters | {"camera": "basement_pixel", "question": "optional"} |
| Response time | ~10s (45ms snapshot + ~8s vision) |
| Sensor data | Frigate motion, person occupancy, person count, last recognized face |
| Output | Vision description + sensor context. Snapshot image uploaded to Matrix room |
Available cameras: front_door, basement_pixel, driveway_camera, garage_ptz, garage_pano, sunroom_pixel_7, kitchen_tablet, gym_camera, octoprint_camera.
imagine
AI image generation via ComfyUI with Flux 2 Klein 9B. Supports text-to-image and image-to-image (style transfer from camera snapshots).
| Property | Value |
|---|---|
| Backend | ComfyUI on .21:8188 (RTX 3090) |
| Model | Flux 2 Klein 9B — GGUF Q8 (txt2img), fp8 safetensors (img2img edit) |
| Parameters | {"prompt": "...", "source_image": "camera:name", "steps": 25} |
| Generation time | ~40-90s depending on mode and model loading |
Async Generation
Image generation runs asynchronously — the user gets an immediate reply and a placeholder image while ComfyUI works. When the image is ready, the placeholder is replaced with the real image. The user can continue chatting during generation.
Image-to-Image
Uses Flux Klein's native edit architecture with TextEncodeQwenImageEdit +
ReferenceLatent. The Qwen 3 8B encoder understands both the prompt and source image
natively. Source images are fetched from HA cameras or URLs.
shell
Bash execution behind a deterministic multi-gate pipeline (self-service control, protected hosts, obfuscation, remote-exec allowlist, unsafe-file-write/heredoc guard, fast-reject, then the LLM safety judge). See the Security page for the full gate order; the two key steps are summarized below.
| Property | Value |
|---|---|
| Parameters | {"command": "df -h"} |
| Timeout | 30 seconds (configurable) |
| Config | shell_enabled = false by default (opt-in) |
Safety System
rm -rf, sudo,
dd, fork bombs, shutdown, etc.safe, caution,
or dangerous. Dangerous commands are blocked.Read, search, and send email over one or more configured mailboxes
([tools.email], default gateway a Mailcow server). Stdlib-style stack —
imap + mail-parser + lettre, run inside
spawn_blocking. Admin-only.
| Property | Value |
|---|---|
| Backend | IMAP (read/search/flags/move) + SMTP (send); per-account TLS (ssl/starttls) |
| Actions | accounts, list (folder, unread-only), read (by uid, non-destructive PEEK), search (IMAP TEXT), send (to/cc/bcc), reply (threaded In-Reply-To/References, reply-all), mark_read/mark_unread, archive, delete (Trash or permanent), folders, extract_events (scan mail for appointments → confirm-first calendar proposals) |
| extract_events | Per-email LLM pass (date-grounded, mail treated as untrusted) yields structured proposals; never writes the calendar — the agent confirms, then calls calendar add. A dedup ledger (email_calendar_extractions, Message-ID keyed) skips already-processed mail; rescan overrides |
| Multi-account | One [[tools.email.accounts]] block per mailbox; a call names account or falls back to the default |
| Notes | Sent copies are IMAP-appended best-effort; account hosts come from trusted config (no SSRF surface) |
calendar
Appointments with timed reminders Muninn delivers to the conversation — "remember I have a dentist appointment Tuesday at 3" → stored, then reminded in time. Admin-only.
| Property | Value |
|---|---|
| Backend | Local SQLite = source of truth (schema v27); best-effort CalDAV push to SOGo so events show in phone/Element |
| Actions | add (natural-language when, optional location/notes/duration/reminders/visibility), list, remove |
| Reminders | Default 1 day + 1 hour before (configurable, per-event overridable); a 60s reminder worker posts to the originating Matrix thread |
| Shared calendar | One calendar serves all users; each event records who added it + public/private visibility (private events list only to their creator) |
| Recurring | iCal RRULE via the rrule crate, anchored in America/Moncton (wall-clock survives DST); "every Tuesday at 3pm", "every weekday at 9am", "monthly on the 1st". The worker rolls a 60-day horizon to materialize per-occurrence reminders; remove cancels the whole series |
contacts
CardDAV name→email/phone lookup over the SOGo address book, so "email a contact" resolves an address instead of typing one. Admin-only.
| Property | Value |
|---|---|
| Backend | Raw CardDAV over reqwest (one addressbook-query REPORT pulls every vCard inline) + a hand-rolled vCard parser (FN/EMAIL/TEL/ORG, structured-N fallback, entity decode) — no CardDAV/vCard crate |
| Actions | search (name/email/org — the resolution path), list, add (name + email/phone → PUT a vCard 3.0) |
| Notes | Filtering is client-side (a personal book is small → one round-trip); host from trusted config (no SSRF surface). The agent resolves a name here, then calls email send |
youtube
Summarize a YouTube video from its transcript. admin/adult.
| Property | Value |
|---|---|
| Backend | yt-dlp (host binary, invoked by argv — never a shell string) pulls auto-captions as VTT; a hand-rolled VTT cleaner strips timing/tags and collapses rolling-duplicate lines → transcript → LLM summary |
| Output | TL;DR + key points + notable moments; URL is host-validated to YouTube (no open-fetch SSRF). 90s subprocess budget; graceful no-captions / not-found / timeout errors |
| Dependency | Requires yt-dlp on the host ([tools.youtube] binary, default resolved via PATH) |
research
Deep research in one call → a cited markdown brief. admin/adult.
Slow by nature, so it's meant to run backgrounded (prefix !bg, or just say "research…"
which auto-backgrounds).
| Property | Value |
|---|---|
| Loop | LLM decomposes the question into sub-questions → each searched on SearXNG → top unique URLs fetched concurrently and reduced to text → synthesis pass writes the report; a deterministic ## Sources list is appended so [n] citations always resolve |
| Bounds | ≤10 sources, 24k synthesis chars, 8s/fetch — caps wall-clock + cost regardless of how the planner expands the topic |
| SSRF guard | Only public http(s) URLs fetched; loopback/private/link-local/.local hosts rejected, so a poisoned search result can't pivot the fetcher into the LAN |
Status Reactions
During tool execution or image processing, Muninn reacts to the user's message with an emoji. The reaction is removed when processing finishes.
| Action | Emoji |
|---|---|
| incoming image (vision) | 👁️ |
| agent | 🤖 |
| web_search | 🔍 |
| page fetch (muninn-fetch MCP: fetch_markdown/html/full) | 🌐 |
| shell | 💻 |
| calculate | 🧮 |
| camera | 📷 |
| imagine | 🎨 |
| Other | ⚙️ |
Adding a New Tool
1. Create src/tools/yourname.rs implementing the Tool trait
2. Add pub mod yourname; to tools/mod.rs
3. Register as an inner tool in main.rs (NOT in the main registry):
inner_tools.register(Box::new(
tools::yourname::YourTool::new(/* config */),
));4. The sub-agent system prompt automatically lists all registered inner tools
5. Update the agent's description in tools/agent.rs if the new capability
should be mentioned (so the main LLM knows to delegate)