this post was submitted on 24 Jun 2026
113 points (83.4% liked)
Selfhosted
60409 readers
272 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil.
-
No spam.
-
Posts are to be related to self-hosting.
-
Don't duplicate the full text of your blog or readme if you're providing a link.
-
Submission headline should match the article title.
-
No trolling.
-
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads "sparse" parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
Since implementation of the
--fitparameter and its relatives, and--fit onbecoming the default, llama.cpp intelligently decides what to offload. For me, it made--n-cpu-moeobsolete.Mostly, yeah.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap, and don’t OOM on the GPU either.
You can control how much context should be fitted with
--fit-ctxand how much space the algorithm should leave unallocated (even on a per-GPU basis) with--fit-target.