this post was submitted on 21 Jul 2026
25 points (100.0% liked)
Technology
1469 readers
22 users here now
A tech news sub for communists
founded 4 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
DeepSeek V4 flash is feasiblish to run it personally. A minipc costing $3000 can do a working quantization, two such computers can run what is in practice a full model. You could even run a (weak but working) quantization of GLM5.2 in this configuration.
What you get running models this way is control over what exactly runs (you never know how aggresively quantized are models running with API, for example) and possibility of running anything that fits (e.g. models with refusals removed with methods like heretic).
What you lose is that you need to buy special hardware for thousands of dollars (which might or might not age well) and loading prompt (including the history of conversation) can be really slow compared to what bigger machines that host models for many users can achieve. Loading past context of 100,000s tokens takes actual minutes while using the same model through API will feel instant.
It shows how important is for people to own the datacenters. Running model on bigger hardware for multiple people is much more efficient than running it on personal hardware.
$3000 in today's prices or of six months ago?
Current prices, I put example under the neighbour comment.
Would you able to please spec such a PC?
https://www.bosgamepc.com/products/bosgame-m5-ai-mini-desktop-ryzen-ai-max-395 it's still below $3K. It's not the optimal Strix Halo variant if you want to cluster a few of them though (you still can use thunderbolt in this one to connect it to another one).
Thank you! I had no idea!
(I was curious - wanted to have an idea what would be considered feasible these days but personally would rather pay for the tokens if needed as you suggested. I am however going to use the specs of the models you suggested to learn more about how AI works in such a machine)
they'd be pretty slow though
About how strong these models are compared to claude - DeepSeek flash is much better than Haiku, GLM 5.2 is comparable at level of Opus, definitely better than Sonnet. In a quantization you could run with 256GB of VRAM, it would still be better than Sonnet I think.
You can run these models with API and save a lot of money compared to Claude too if you swap: Haiku -> DeepSeek Flash, Sonnet -> DeepSeek Pro, Opus -> GLM 5.2, Fable -> Kimi K3
Which also is why the AI companies are panicking a little bit. They don't necessarily target private consumer anyways, but are trying to expand into the business-licensing space. Thats why they are competing so hard to create the most complete ecosystem of tools, i.e. coding, legal work, picture+movie, defense.
But no reasonably lead company is going to waste $9999999 billions on tokens for ever more expensive models when you can get similar results by expanding your serverhall with a few m^2.
Interesting...
Can you tweak or change the AI completely?
Or even somehow notifies revamp it into your own AI completely?
Maybe build off of it somehow and improve it maybe?
If you mean using different models, yes. If you mean finetuning model weights... No, finetuning LLMs is hard in general and Strix Halo (device that I had in mind that costs 3K$ for 128GB of VRAM) is also kinda bad for it now due to having an AMD (and not really strong when it comes to compute power) GPU.
In practice, people who try finetuning on top of newest models usually cripple them. There are plenty of finetunes of Qwen 3.6 on huggingface trying to distill stronger models and they are just bad.
Heretic, the method for removing refusals I mentioned, also requires more VRAM to run it than it's required for inference.
"AMD GPU"
oof