This is an automated archive made by the Lemmit Bot.
The original was posted on /r/homeassistant by /u/TheOriginalOnee on 2026-07-22 07:31:50+00:00.
Since some people on my last post asked for the setup details, here is a write up of how exactly my HA LLM + STT is set up.
I run Gemma 4 12B Q8_0 locally on an NVIDIA RTX 2000 Ada Generation GPU with exactly 16 GB VRAM. The A2000 Ada was selected speciffifally for its very low stand by power usage of around 5W and its also low inference power draw of 70W. Its performance is sufficient for small models like gemma 4 12B in my usecase.
The same Gemma instance handles normal LLM usage and German Home Assistant Voice Assist transcription:
llama.cpp setup
I use the CUDA llama.cpp server and configure models through the models.ini file instead of maintaining a large startup command.
The active model artifacts are all from Unsloth's unsloth/gemma-4-12b-it-GGUF release family:
- Main model: gemma-4-12b-it-Q8_0.gguf
- Multimodal projector: mmproj-gemma-4-12B-it-bf16.gguf
- MTP draft model: mtp-gemma-4-12b-it.gguf
The mmproj file is mandatory. It enables Gemma's multimodal input path, including audio transcription. Without it, this is only a text model.
My active preset:
[Gemma4-12B-Q8_0]
model = /models/gemma-4-12b-it-Q8_0.gguf
mmproj = /models/mmproj/mmproj-gemma-4-12B-it-bf16.gguf
model-draft = /models/mtp-gemma-4-12b-it.gguf
spec-type = draft-mtp
spec-draft-n-max = 3
n-gpu-layers = 999
flash-attn = on
parallel = 1
batch-size = 256
ubatch-size = 256
cache-type-k = q8_0
cache-type-v = q8_0
ctx-size = 140000
jinja = true
reasoning = off
Performance on 16 GB VRAM
This configuration supports up to 140,000 tokens of context in my setup. Your actual limit depends on driver, llama.cpp version, request size, and other models loaded on the GPU, so monitor VRAM usage yourself.
For most tasks, 140k context is okay.
The settings that matter most are:
- Full GPU offload through "n-gpu-layers = 999"
- Flash Attention enabled
- Q8 KV cache
- One parallel model request with "parallel = 1"
- MTP speculative decoding
The matching Unsloth MTP draft model made one of the biggest practical differences to response speed. llama.cpp can validate multiple predicted tokens at once instead of generating every token sequentially. This noticeably improves normal LLM response latency even on a relatifly weak GPU.
Disable reasoning
This is essential when the same Gemma model is also used for STT:
reasoning = off
It must be disabled in the llama.cpp "models.ini" preset itself. Otherwise reasoning is also enabled for transcription requests.
Without this setting, Gemma can generate unnecessary reasoning or explanation tokens around short voice commands. That makes STT slower and less reliable.
Googles Update for Gemma
Gemma is very capable across languages. German works well for both normal conversation and speech transcription in my experience, but for tool calling it had some problmes in the past.
Google recently released an updated official Gemma chat template. The active Unsloth Q8_0 GGUF now also includes this update, and llama.cpp uses the embedded template through:
jinja = true
This matters for tool calling. The updated official template formats tool definitions and tool responses more reliably than older versions.
I would not blindly add an old external Jinja template file. Make sure your GGUF contains the current official template and that llama.cpp uses the embedded version.
Home Assistant STT
Home Assistant uses the Wyoming protocol, while llama.cpp provides an OpenAI-compatible transcription endpoint.
I connect both with: ghcr.io/roryeckel/wyoming_openai
The request path is:
Home Assistant -> Wyoming OpenAI -> llama.cpp /v1/audio/transcriptions
The Wyoming container does not need GPU access.
Relevant container settings:
- Restart policy: unless-stopped
- WYOMING_LANGUAGES=de
- STT_OPENAI_URL=http://<llama-server>:<port>/v1
- STT_BACKEND=OPENAI
- STT_TEMPERATURE=0
The most important STT setting is:
STT_TEMPERATURE=0
My normal LLM profile uses more creative sampling, but STT should be deterministic. Without this per-request override, Gemma sometimes transcribed correctly and then continued with a translation, explanation, or answer to the spoken question.
I use this transcription prompt:
Task: German speech recognition. Return only the spoken German text verbatim. Do not translate, explain, summarize, answer, format, or add labels. If there is no intelligible speech, return an empty response.
The prompt is passed as the OpenAI transcription "prompt" field, not as a normal chat system prompt.
Home Assistant setup
- Add the Wyoming Protocol integration.
- Point it to the Wyoming-OpenAI container.
- Select the discovered Gemma STT engine in the Assist pipeline.
- Set the Assist pipeline language to German.
Avergate Timings are:
0,8 Sec for STT
2-3 Sec for LLM response with (multi) tool use
TTS
This setup has no local TTS.
Gemma does not generate speech in this deployment, and a 16 GB GPU running Gemma 4 12B Q8_0 has no meaningful VRAM left for a separate high-quality local TTS model.
I would keep TTS separate and use ElevenLabs for natural German voices. That keeps the GPU focused on Gemma for LLM and STT while providing better speech output quality.
Summary
A 16 GB GPU is enough for Gemma 4 12B Q8_0 as a local LLM and Home Assistant STT backend when configured carefully:
- NVIDIA RTX 2000 Ada Generation with 16 GB VRAM
- Unsloth Gemma 4 12B Q8_0 GGUF
- Matching Unsloth BF16 `mmproj`
- Matching Unsloth MTP draft model
- Full GPU offload
- Flash Attention
- Up to 140k context
- MTP speculative decoding
- One parallel request
- Current embedded official chat template
- `reasoning = off`
- `STT_TEMPERATURE = 0`
Sorry for the long post. AI was used for translation.
