Voice control in the home has an unforgiving latency and reliability bar. When someone walks into a dark kitchen carrying groceries and asks to turn on the counter lights, they expect the lights to come on in under a second. They do not want to memorize exact device names, and they do not want to wait six seconds while an AI agent reasons through tool schemas.
Most self-hosted smart-home voice pipelines get trapped between two flawed options:
- Rigid local intent matchers. Home Assistant’s built-in sentence matcher (
conversation.home_assistant) is fast and free, but brittle. It requires exact phrase templates and canonical entity names. If a family member says “kill the office heater” instead of “turn off basement heater,” or if a local Whisper speech-to-text pass transcribes “shut off the air philter” or “paws the kichen speaker,” the matcher fails. - Full LLM agent loops. Routing every failed match to a general-purpose tool-calling agent fixes comprehension at the cost of speed and token spend. Loading a household device inventory and tool definitions into context takes 1,200 to 15,000+ tokens per turn and adds four to eight seconds of latency just to flip a switch.
We wanted a third path for our Home Assistant voice pipeline: a sub-second semantic fast path that understands messy human speech, room context, and Whisper transcription noise, executes physical device actions and live state queries safely, asks for clarification when a command is genuinely ambiguous, and escalates open-ended questions (“is it going to rain before soccer practice?”) to our full assistant without delay.
Using Fireworks’ Serverless Training platform, we fine-tuned a Muse Glimmer 30B LoRA adapter on a synthetic household voice corpus, paired it with a deterministic entity resolver, and deployed it behind our Wyoming voice bridge. On our 89-example holdout benchmark, the tuned adapter improved end-to-end accuracy from 24.0% (stock zero-shot with the full ontology in prompt) to 98.9% at a 503-millisecond median sampler latency, with zero wrong physical actuations and an operating cost of roughly $0.42 to $0.71 per month for a busy household.
Architecture: Decoupling Semantic Intent From Physical Hardware
When teams first consider fine-tuning a model for smart-home control, the temptation is to train the model to emit raw Home Assistant service payloads directly:
{"domain": "switch", "service": "turn_off", "entity_id": "switch.basement_heater_socket_1"}
That shortcut creates an operational headache the moment your hardware changes. Smart homes drift constantly. A Zigbee smart plug dies and gets replaced with a Matter dimmer; a holiday light switch gets repurposed as a patio fan in the spring; a firmware update renames media_player.living_room_2 to media_player.living_room_sonos. If physical entity IDs are baked into your LoRA weights, every hardware swap silently breaks voice control until you curate new training data and retrain the model.
To make a fine-tuned voice model durable in production, you have to separate what the human means from how Home Assistant addresses the hardware today.
Drawing on recent open-source and practitioner work on small Home Assistant voice models, including Selora-AI-LLM-1.7B, infohound/ha-voice-granite-3b-GGUF, Pau Labarta Bajo’s fine-tuning workflow with Liquid AI’s Home Assistant recipe, Big Iron’s voice latency budget analysis, TechFuelHQ’s 2026 local voice tutorial, and community voice agent designs in r/homeassistant and Pucciano/ha-voice-agent, we structured the fast path around two layers and a compact four-route JSON schema:
- A learned semantic layer (Muse Glimmer 30B LoRA). The model takes the microphone’s room tag and raw transcript (
[mic: downstairs] shut off the air philter) and emits a compact JSON envelope with one of four single-character route keys:c(Direct command or multi-command chain):{"c": [["downstairs", "air_purifier", "turn_off"]]}or{"c": [["living_room", "main_lights", "turn_off"], ["living_room", "tv", "turn_on"]]}q(Read-only live state query):{"q": [["downstairs", "basement_heater"]]}for questions like “what is the office heater set to?” or “is the garage door closed?”k(Immediate ambiguity clarification):{"k": "Which heater: Charlie's or Freddie's?"}when a command targets a room with multiple devices of the same class or comes from a device without a room tag ([mic: unknown]).e(Escalation to the full assistant):{"e": "weather_or_external_info"}for open-ended questions, timers, or multi-step reasoning.
- A deterministic resolution and speech layer (
ontology.json+resolver.py). A lightweight Python resolver validates that every predicted(area, target, action, value)tuple is legal, maps canonical targets to live Home Assistant entity IDs, formats live sensor/attribute readings forqqueries, and synthesizes natural spoken confirmations deterministically in zero milliseconds.
That boundary changes how the system handles hardware drift across our 8 rooms and 26 controllable device targets:
| Change in the Home | Share of Real-World Drift | Impact on LoRA Weights | Required Update |
|---|---|---|---|
| Replace a smart plug, rename an entity ID, or swap a switch for a dimmer | ~80% | None. Semantic target (downstairs.basement_heater) is unchanged. | Edit one line in ontology.json; reload in 1 second. |
| Add a new spoken nickname for an existing device | ~10% | Minor until next retrain; existing synonyms still resolve. | Append alias to ontology.json and include in next scheduled refresh. |
| Add a brand-new room or a new device class | ~10% | Model does not know the new target slug yet; safely escalates or rejects via resolver. | Run synthetic data generator + 8-minute LoRA retrain (~$1.53). |
Crucially, the deterministic resolver acts as a hard safety rail. If the model ever hallucinates a target that does not exist in ontology.json, pairs a target with the wrong room, or tries to call set_temperature on a binary switch, resolver.py rejects the intent before touching Home Assistant and falls back to the conversational assistant.
Why Output Token Count Matters on a Serverless Sampler
Both Selora-AI-LLM-1.7B and Liquid AI emphasize keeping structured voice outputs as short as possible and synthesizing spoken confirmation text in code rather than generating it token-by-token with the LLM.
To verify how much output token length actually affects wall-clock latency on Fireworks’ serverless sampler (versus fixed HTTP and prefill overhead), we benchmarked four output envelope lengths (N=12 prompts each after warmup) against the live FireTitan endpoint:
| Envelope Shape | Output Tokens | Latency p50 | Latency Mean | Latency p95 | Delta vs. Verbose + Reply (38 tok) |
|---|---|---|---|---|---|
1. Verbose JSON keys + model-generated "reply" | 38.2 tok | 780.9ms | 797.5ms | 917.2ms | baseline |
2. Verbose JSON keys without "reply" | 26.8 tok | 638.9ms | 661.0ms | 720.1ms | -142.0ms (-18.2%) |
3. Slim tuple envelope ({"c":[["area","target","action",val]]}) | 18.2 tok | 503.0ms | 552.7ms | 629.5ms | -277.9ms (-35.6%) |
| 4. Ultra-minimal floor probe | 12.0 tok | 495.5ms | 484.3ms | 532.9ms | -285.4ms (-36.5%) |
Linear regression across all 48 post-warmup samples showed a fixed network and prefill floor of 337.9ms and a marginal decode cost of 12.18ms per output token (~82 tokens/sec). Moving the spoken confirmation string into resolver.py and using the slim c/q/k/e tuple envelope cut output tokens in half (38.2 to 18.2 tokens) and shaved 278ms off median inference latency (-35.6%).
Generating a Grounded Voice Corpus: Coverage Audits and Gateway Spend
A domain adapter is only as good as the distribution of its training data. Writing hundreds of voice commands by hand usually produces clean, repetitive sentences that look nothing like what comes out of a wall microphone and an edge speech-to-text model.
Instead of hand-writing prompts, we treated ontology.json and resolver.py as a formal specification and used a frontier teacher model through our LiteLLM gateway to synthesize a diverse corpus across all 26 device targets and 8 rooms. Every generated row had to pass resolver.py validation before entering the dataset.
Following Pau Labarta Bajo’s Home Assistant SLM workflow and Liquid AI, we also added a programmatic coverage gate (audit_action_coverage()) that walks every (area, target, action) triple in ontology.json and fails the build if any capability has fewer than three training examples. Our final 853-example dataset (764 training rows and 89 stratified holdout rows, with zero normalized-utterance overlap between splits) covers 108 out of 108 valid device actions across the house.
Because every batch call and coding-agent turn flows through LiteLLM into Postgres (LiteLLM_SpendLogs) and our self-hosted Opik trace store (opik.spans), we can measure the exact token volume, wall-clock time, and dollar cost required to generate the corpus from scratch:
Telemetry Source (01:04 to 01:16 UTC) | Calls / Spans | Prompt Tokens | Completion Tokens | Total Tokens | Recorded Gateway Spend |
|---|---|---|---|---|---|
Accepted batch synthesis (usage.jsonl) | 98 calls (96 main + 2 retry) | 37,517 | 81,569 | 119,086 | $0.0009 (~$0.45 at commercial list rates) |
All synthesis calls incl. smoke probes (LiteLLM_SpendLogs) | 101 calls (14.8s avg latency) | 61,135 | 87,647 | 148,782 | $0.0010 (~$0.51 at commercial list rates) |
Opik synthesis trace spans (opik.spans) | 102 spans (15.0s avg latency) | 80,187 | 90,446 | 170,633 | $0.0014 (~$0.55 at commercial list rates) |
| Interactive agent engineering loop (ontology + resolver + runner) | 81 agent turns | 8,504,094 | 41,793 | 8,545,887 | $0.7501 ($0.664 frontier + $0.086 fast tier) |
| Total end-to-end build + corpus synthesis | 183 calls | 8,584,281 | 132,239 | 8,716,520 | $0.7515 total |
Running the 96-task synthesis batch (with 5 concurrent workers) took 4 minutes and 38 seconds, plus 20 seconds for a 2-task retry pass. The raw synthesis tokens cost $0.0010 to $0.0014 on our gateway (or roughly $0.45 to $0.55 if billed at standard commercial frontier API list prices), and the entire end-to-end agent session that inspected our Home Assistant inventory, wrote the ontology and resolver, and generated the corpus cost $0.75.
The dataset is stratified across six real-world voice patterns:
- Household aliases and informal phrasing: Mapping “office heater” or “downstairs space heater” to
downstairs.basement_heater, “counter lights” tokitchen.under_cabinet_lights, and “foyer lamp” toliving_room.entryway_light. - Whisper ASR phonetic corruptions: Realistic homophones and transcription artifacts with no punctuation, such as “shut off the air philter” (
air_purifier), “paws the kichen speaker” (kitchen_speaker), “turn on the dining Chandelier” (chandelier), and “set bedroom light to fit tee percent” (50%). - Room-context defaults and multi-action chaining: Short commands where the user omits the room name because they are speaking into a room-specific satellite microphone (
[mic: master_bedroom] heater on), as well as compound requests ([mic: living_room] turn off the main lights and turn on the tv). - Read-only live state queries (
q): Questions about current device state and attributes, such as “is the garage door closed right now?” or “what is the office heater set to?” (infohound/ha-voice-granite-3b-GGUF). - Ambiguous commands requiring clarification (
k): Commands like[mic: kids_room] turn on the heater(where Charlie and Freddie each have a heater) or[mic: unknown] turn on the heaterfrom a mobile device (r/homeassistant, Pucciano/ha-voice-agent). - Hard-negative escalations (
e): Prompts that sound like home commands on the surface but require external tools, timers, or multi-step logic, such as “turn off the fireplace in 20 minutes,” “is it going to rain before soccer practice,” or “why is the internet slow in the office?”
Benchmarking Stock Muse Glimmer 30B vs. the Tuned LoRA
Before deploying the adapter to our voice bridge, we wanted an honest answer to a simple question: is a fine-tuned LoRA actually better than stock Muse Glimmer 30B if we just put the full household device ontology into the system prompt?
To test that, we evaluated stock muse-glimmer-30b across three prompting and reasoning configurations (compact prompt without thinking, full 1,195-token ontology prompt without thinking, and full ontology prompt with up to 512 internal reasoning tokens) and compared them against our fine-tuned LoRA adapter (trained for 2 epochs / 48 steps on FireTitan Serverless at $0.00 promotional training cost):
| Evaluation Metric | 1. Stock Compact (No-Think) | 2. Stock + Full Ontology (No-Think) | 3. Stock + Full Ontology (Think 512) | 4. Tuned LoRA (Compact, No-Think) |
|---|---|---|---|---|
| Avg Prompt / Output Tokens | 117 / 23 tok | 1,195 / 23 tok | 1,192 / 320 tok | 117 / 20 tok (18 tok on commands) |
| Valid JSON Rate | 100.0% | 100.0% | 82.7% (17.3% truncated) | 100.0% (89 / 89) |
Direct Command Accuracy (c) | 0.0% | 25.7% (18 / 70) | 78.6% (55 / 70) | 98.6% (71 / 72) |
Live State Query Accuracy (q) | 0.0% | n/a (escalated) | n/a (escalated) | 100.0% (7 / 7) |
Ambiguity Clarification (k) | 0.0% | n/a (escalated) | n/a (escalated) | 100.0% (5 / 5) |
Escalation Recall (e) | 100.0% | 100.0% (5 / 5) | 100.0% (5 / 5) | 100.0% (5 / 5) |
| End-to-End Resolver Accuracy | 5.6% (5 / 89) | 24.0% (18 / 75) | 73.3% (55 / 75) | 98.9% (88 / 89) |
| Wrong Physical Actuations | 0 | 0 | 1 | 0 |
Sampler Latency p50 / p95 | 503ms / 630ms | 772ms / 3,891ms | 4,415ms / 7,903ms | 503ms / 630ms |
The comparison shows why neither zero-shot prompting nor test-time reasoning solves the voice fast-path problem on its own:
- Without thinking, stock Glimmer cannot bridge phonetic noise or household aliases. When given the full 1,195-token ontology with thinking disabled, stock Glimmer handled literal commands like “set dining room dimmer to 80 percent,” hitting 25.7% direct accuracy. Whenever a prompt included a Whisper homophone (“shut off the air philter”, “paws the kichen speaker”), a household nickname (“set office heater to 72”), or an implicit action (“goodnight lights”), the base model panicked and escalated to the slow assistant.
- With thinking enabled, stock Glimmer gets much smarter but breaks the voice latency budget. Allowing the base model to reason in
to=selflifted direct service-call accuracy from 25.7% to 78.6%. However, generating 320 reasoning tokens pushed median latency to 4.42 seconds (7.90 seconds at p95), and 17.3% of responses ran out of token budget mid-thought before emitting JSON. Even after thinking for four seconds, the base model still missed 21.4% of direct commands on household conventions. - Fine-tuning compiles reasoning and domain conventions into zero-thought weights. Our tuned LoRA achieved 98.9% end-to-end accuracy (88/89) across direct commands (
71/72), live state queries (7/7), clarifications (5/5), and escalations (5/5), with 87.6% exact string match (p = 6.6e-24), zero thinking tokens, and a 90% smaller prompt (117 tokens vs. 1,195 tokens). - Zero wrong physical actuations. On the single holdout miss out of 89 examples (
[mic: living_room] turn on the office lamp), the model predictedliving_room.office_lampinstead of escalating. Becauseoffice_lampis not a valid target inliving_room,resolver.pyrejected the output cleanly and fell back to the assistant without touching any switch.
Unit Economics and Stateless Token-In/Token-Out Serving
Running a custom 30B model for home automation used to mean renting a dedicated GPU instance at $1,500 to $3,000 per month, which makes no sense for a household that issues 50 to 100 voice commands a day.
Because Fireworks’ FireTitan Serverless Training platform exposes a shared rollout sampler pool (POST /training/v1/serverless/inference/v1/completions), we only pay for the exact tokens processed when someone speaks into a microphone. Idle hosting cost is $0.00 per hour.
One architectural detail is worth calling out for teams building on FireTitan: serverless LoRA sampling uses client-side tokenization (token-in, token-out, or TITO) rather than server-side chat templates, and it does not require keeping a persistent training loop running.
On Fireworks, muse-glimmer-30b LoRA adapters are stored as HF_PEFT_ADDON models with supports_serverless: false on the public /inference/v1/chat/completions router. To serve them without paying for a dedicated on-demand GPU deployment:
- One-time snapshot registration: A lightweight control-plane session (
ts-<hex>) callsload_adapter(promoted_model)andsave_weights_for_sampler()once to publish a hotloadable LoRA snapshot into the shared FireTitan rollout pool, then immediately closes the training client handle. - Stateless HTTP inference with local tokenization: At runtime, our local bridge holds no active trainer connection. It loads the HuggingFace
meta-models/Muse-Glimmer-30Btokenizer andMuseGlimmerRendererlocally, renders[mic: area]and the transcript into raw ATEM token IDs (prompt: list[int]), and sends a stateless HTTP POST to/training/v1/serverless/inference/v1/completions(raw_output=True), decoding the returned completion token IDs in Python.
At Fireworks’ published Muse Glimmer 30B serverless rates ($1.96 / 1M uncached input tokens, $0.39 / 1M cached input tokens, and $4.88 / 1M output tokens), our compact 117-token prompt and 19.8-token JSON output make the fast path virtually free to operate:
| Usage / Lifecycle Metric | Token Footprint | Uncached Cost | Prefix-Cached Cost |
|---|---|---|---|
| Single voice command (Tuned LoRA) | 117 prompt + 18 output | $0.000317 (~3,150 / $1) | $0.000182 (~5,490 / $1) |
| Single voice command (Stock + Ontology + Think) | 1,192 prompt + 320 output | $0.003898 (~256 / $1) | $0.002026 (~493 / $1) |
| Light household (30 commands / day) | ~122K tokens / month | $0.29 / month | $0.17 / month |
| Typical household (75 commands / day) | ~304K tokens / month | $0.71 / month | $0.42 / month |
| Heavy household (200 commands / day) | ~810K tokens / month | $1.90 / month | $1.11 / month |
| Full LoRA retrain (764 rows, 2 epochs) | ~245K training tokens | $0.00 (promo thru Nov 30, 2026) | ~$1.53 per retrain after promo |
Compared with stock Glimmer using in-context ontology and thinking, the fine-tuned LoRA is 11x cheaper per call and nearly 9x faster at the median (503ms vs. 4,415ms). Compared with falling back to a multi-turn frontier agent loop ($0.01 to $0.05 per voice command), the fast path cuts per-command cost by 50x to 250x while turning a multi-second pause into an immediate physical response.
Deploying Into the Live Wyoming Voice Pipeline
We wired the model and resolver into our local FireTitan bridge (/v1/ha_voice) and placed it directly ahead of the LLM fallback in our Wyoming WebRTC voice bridge:
- Exact local match first (0ms, $0): If Home Assistant’s native sentence matcher resolves an exact phrase, it executes immediately.
- Glimmer 30B LoRA + Resolver fast path (~500ms, $0.0002): Whenever the built-in matcher misses or appears to hallucinate an answer on a device command, the bridge sends the transcript and satellite room area to
/v1/ha_voice.- Direct commands (
c): Ifresolver.pyvalidates the service payload, the bridge executes the Home Assistant service calls directly and speaks the synthesized confirmation ("Turning off the basement heater."in ~570ms). - Live state queries (
q): If the user asks for device status (“what is the office heater set to?”, “is the garage door closed?”), the resolver formats a spoken answer directly from live Home Assistant state and attributes ("The basement heater is off and the room is 72 degrees."in ~505ms). - Ambiguity clarifications (
k): If a command targets an ambiguous device (“turn on the heater” in a dual-heater room or without a room tag), the bridge immediately speaks the clarification prompt ("Which heater: master bedroom, office, Charlie's, or Freddie's?"in ~590ms). - Escalations (
e): For open-ended questions, timers, or diagnostics, the bridge skips local regex retries and routes straight to our conversational assistant. Following Big Iron and TechFuelHQ, we also refactored the fallback LLM prompt so itssystemmessage remains 100% static and dynamic clock/sensor context is attached to theuserturn, preserving prefix cache hits on escalated queries.
- Direct commands (
- Closed-loop telemetry: Every fast-path call logs the raw Whisper transcript, microphone area, LoRA output, resolver validation status, and latency to a local JSONL audit log so any new household slang or hardware additions can be folded into the next 8-minute retrain.
Fine-tuning does not make sense for every routing problem. When policy rules fit in two sentences of plain English, a zero-shot classifier is simpler to operate. But when a latency-critical interface has to map noisy, real-world speech onto a structured domain inventory every single time, baking that ontology into a small serverless LoRA gives you the accuracy of a reasoning model at the speed and cost of a lookup table.