this post was submitted on 14 Jul 2026
-6 points (40.0% liked)

LocalLLaMA

4916 readers
12 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago
MODERATORS
pax@sh.itjust.works
noneabove1182@sh.itjust.works
Smokeydope@lemmy.world
MonsterBug@sh.itjust.works
-6
I gave my local LLM agent a face — an avatar frontend with an "emotion-beat" output contract (open-source) (sh.itjust.works)
submitted 6 days ago* (last edited 6 days ago) by catch88@sh.itjust.works to c/localllama@sh.itjust.works
16 comments fedilink hide all child comments
 

I got tired of keeping a Telegram tab open all day just to talk to my local agent, so I built a monitor-resident avatar that fronts it instead — a video-call-style face that performs the replies instead of me reading a wall of text.

A few things that might interest this crowd:

No runtime GPU for the avatar. Instead of live-animating a face, the agent's reply carries inline emotion-beat tags — [working], [confirm] deploy?, [happy] — and a player selects and blends between ~30 pre-rendered clips. The GPU stays 100% on the model. Code and logs render as chat cards instead of being read aloud, and [confirm] pops a human-in-the-loop approve/cancel that blocks the agent until you answer.

It hooks into a real agent as a connector. The agent's gateway dials out to a local WebSocket the app hosts, so from the agent's side the avatar is just "another channel," indistinguishable from a messenger. Adapters for Hermes and OpenClaw are included, plus a demo mode that runs with zero setup.

Local voice both ways. Edge TTS out (swappable to Qwen3-TTS / MeloTTS / Piper), Silero VAD + faster-whisper in.

Open-core, MIT. The engine is fully usable on its own: drop in a folder of clips named by emotion, point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own. The repo ships the engine only; there's no bundled character.

The engine just plays video clips — it doesn't care what produced the pixels. So the avatar can be photoreal, 2D anime, a 3D render, pixel art, anything. You can even use a Live2D or VRM model: pre-render its expressions into clips and it becomes a Ghost Vessel preset. A Live2D shell can't do the reverse — it can't take live-action footage. Clips are a superset. The trade-off is that you give up real-time rig interactivity (no dragging the head around) and you pay a one-time render step.

Repo + 45s demo: https://github.com/ghdtjrtka/ghost-vessel

The part I'd most like feedback on is the emotion-beat output format — the contract that turns model text into a UI performance. Has anyone here tried similar output contracts to drive an interface from LLM output? Curious what broke on your setup, and whether smaller local models keep the tags straight.

top 16 comments
sorted by: hot top controversial new old
[–] db2@lemmy.world 10 points 6 days ago (1 children)

[–] HelloRoot@lemy.lol 2 points 6 days ago (1 children)

where is this from?

[–] db2@lemmy.world 2 points 6 days ago (1 children)

Max Headroom, awesome show from the 80s. But be warned, it did not age well.

[–] scholar@lemmy.world 1 points 2 days ago

Also awesome 1985 film from the UK

[–] thebardingreen@lemmy.starlightkel.xyz 2 points 4 days ago

I have one that stitches animations together from several different starting frames A->B, B->C, C->A. I generated a bunch of First Frame / Last Frame videos and randomize them in sequence. It uses MuseTalk for lip sync when it's talking. It looks much more natural.

Mine is a cartoon scientist in a lab. When she's working on stuff, in the background (I put her in loops all the time, where she's just heads down on some task), she like, mixes chemicals and messes with electronics and rummages in drawers and eats pizza. When she's in conversation mode, she loops through the same set of starting and ending frames, but is actually passively looking at the camera, so it looks like she's talking to you (or creepily staring at the camera for hours at a time). I can tell if she's finished a /lab or /troubleshoot loop just by checking her animations.

[–] WhiteRice@lemmy.ml 4 points 6 days ago (1 children)

Op, why are they hot?

[–] schwim@piefed.zip 5 points 6 days ago (1 children)

Such an interesting project but I think that would creep me out too much to use it.

[–] catch88@sh.itjust.works 5 points 6 days ago

Ha, that's completely fair — a photoreal face watching you work all day is a lot, and it was honestly the first thing I second-guessed about my own setup. Worth saying though: that's the preset, not the engine. It just plays video clips and doesn't care what produced the pixels, so an avatar can be 2D anime, a 3D render, pixel art, an abstract shape, a blinking terminal face, a cat — whatever's least uncanny for you. The photoreal one is just what I happened to build for myself. (You can also hide the video window and run it chat-only, though at that point you've reinvented a chat window.) But yeah, the uncanny thing is real. Photoreal was the hardest default to justify and I'm still not fully sure it's the right one.

[–] hendrik@palaver.p3x.de 2 points 6 days ago* (last edited 6 days ago) (2 children)

Some 4 or 5 nice default avatars would be helpful. So we can just run it and try, without going through an entire ordeal. And maybe some Colab notebook or pre-made script to create those looping clips, I mean they should be seamless and all and that requires a bit of extra effort, not just "use your favorite tool".

[–] catch88@sh.itjust.works 4 points 6 days ago (1 children)

Fair hit — "loop-cut the segments" was doing a lot of hiding in that sentence. Seamless is the whole difficulty, and "use your favorite tool" was a cop-out. So I've pushed the tooling I'd been using privately:

  • build_idle_loop.py — runs MediaPipe over the take, measures eye-openness per frame, finds the blink minima, then picks the blink→blink window that maximizes eyes-open time and pose match. The cut point lands on a closed eye, so the loop's seam is invisible. Pingpong fallback when there's no clean blink pair, plus it records the head-frontal "settle times" the player uses to time expression reveals.
  • cut_emotions.py — contact sheet to spot the neutral valleys in a multi-emotion take, then cuts and web-encodes them into per-emotion segments.
  • validate_preset.py — PASS/FAIL gate on structure, assets, and emotion coverage.

Plain Python + ffmpeg + MediaPipe, no notebook yet, but they should run anywhere.

On the default avatars: you're right, and I don't have a good defence. The blocker isn't code — it's that an avatar is a pile of clips someone has to actually render, and right now I only have the one I built for myself. A small free starter preset is the obvious next move; I just haven't made it. Until then it's genuinely "bring your own," which is a fair thing to hold against it.

[–] hendrik@palaver.p3x.de 1 points 6 days ago* (last edited 6 days ago)

Yeah. Just add creating the example avatars to the project's ROADMAP or TODO list. It'd be nice to have that from user perspective, just to get started. And after that people can create their own avatars. But you need to make the first step easy for them.

And if I might add: Skip the nonsense with the different licenses. Currently you have a correct LICENSE file with the MIT license. But then also a NOTICE file which says the MIT license doesn't apply to all the project. And a different notice with different wording in the README.md. That's kinda convoluted. And unnecessary complex since you don't even provide avatars and/or presets... Just do the entire project under MIT license. Batteries included. That'll aid with distribution.

[–] catch88@sh.itjust.works 1 points 6 days ago

Small correction to my own comment: the emotion clips were actually made with Gemini i2v — 3 emotions per generation, which is exactly why cut_emotions.py exists (it cuts the take at the neutral valleys).

LivePortrait is the free local route and it does work — the project's earlier expression library was built with it, and it gives you one clip per driving video, so you skip the cutting step entirely. I've fixed the README to state both instead of implying LivePortrait was the path I'd personally taken.

[–] HelloRoot@lemy.lol 1 points 6 days ago

We already have https://wiki.nyarchlinux.moe/nyarchassistant/

[–] Eyekaytee@aussie.zone 1 points 6 days ago (1 children)

I duno if you meant to but the transition effect reminded me of the civ2 advisors: https://www.youtube.com/watch?v=YzE2fpe0kCs

[–] catch88@sh.itjust.works 2 points 6 days ago

Ha — not intentional, but that's a hell of a catch, and honestly the best description of the project anyone's given me. Pre-rendered FMV people who pop up to tell you what's happening… yeah, that's exactly what this is, just with an LLM deciding which advisor shows up. I'll take it as a compliment.