This is a most excellent place for technology news and articles.
This is true. But VLLM's quantization support (as outlined in the article) is quite primitive.
Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.
If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.
If you want raw speed on a desktop, use one of the ninfer project branches.
inari@piefed.zip: Are you a bot? You are posting multiple times every hour of the day.
Nope. I post things I find on Reddit or Hacker News sometimes, though.
Do you know the recipe of the apple pie?
Nah, I dont think so. There are comments mixed with posts, there arent that many posts, and they seem to be reasonably curated.
If you ascribe any level of sentience or intelligence to a computer, you definitely feel much less dumb than you actually are.
Nice strawman you got there. Is this relevant to the post, or do you just feel like calling random unrelated people dumb?
Having used these extensively, I can say with confidence that they feel exactly as dumb as they are.
Do you mind sharing what models you used and what your experience was? In my opinion the Qwen 3.6 models or maybe the 3.5 were the first local models that were actually useful, but I don’t have that much experience.
Qwen is really good. For coding, it's the best local model I've found.
This really killed a wish to run local LLMs. So fiddly...
You shouldn't let that discourage you. VLLM is really finicky, but there are easier backends to use (though which one depends on your hardware).
Not any more than any other local service. Ollama makes it really easy to get started. The ministral models are small but still useful.
https://sleepingrobots.com/dreams/stop-using-ollama/
And thats just the tip of the iceberg.
Also, while Minstral is fine, it is pretty much obsolete...
Yeah, I'm just saying it's a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better.
Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff.
I think LM Studio would be a much better "1st time" recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to "graduate" from it to croco.cpp or llama.cpp or something.
