this post was submitted on 31 Jul 2026
39 points (100.0% liked)
chat
8640 readers
183 users here now
Chat is a text only community for casual conversation, please keep shitposting to the absolute minimum. This is intended to be a separate space from c/chapotraphouse or the daily megathread. Chat does this by being a long-form community where topics will remain from day to day unlike the megathread, and it is distinct from c/chapotraphouse in that we ask you to engage in this community in a genuine way. Please keep shitposting, bits, and irony to a minimum.
As with all communities posts need to abide by the code of conduct, additionally moderators will remove any posts or comments deemed to be inappropriate.
Thank you and happy chatting!
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I'm curious what model you run and on what hardware?
qwen 3.6 35b a3b, 2k usd custom build computer from 2-3 years ago with about 80gb cumulative vram/ram that was partially paid for by my employer. i do nondescript science™️ for my job. its pretty useful for dredging the internet for sources, especially sources that would be normally inaccessible to me because i dont speak the language the paper is in. my previous computer was falling to bits (nearly 20 years old, i hope i can make this one go that far) so i needed a good one anyways.
How on earth do you have 80 gigs of VRAM?
cumulative, i only have ~12gb vram
Oh, I see. So, I have 44gb cumulative ram, 12gb VRAM and 32gb "regular".
I was hoping there was some magical way to improve my 4070ti (not super).
this model can be compressed to as low as 12 gigs (though I wouldn't recommend going that low due to a huge quality drop). And not all of this kind of model has to reside in vram - you can have most of it in ram and the rest gets cached in vram
I'm curious, can the model fill your whole VRAM or do you have to leave extra space for the actual, you know, whatever it's doing? Like, could you actually do anything with the 12gb model if you only have 12 GB of VRAM?
Yes, you need to leave some space for the context (kv cache) - a lot of space if you need a lot of context. Large context just allows you to generate more text and run longer tasks.
12 gigs is actually a good amount to run this class of model. As I said, not all of the model has to live in the vram
here is an example of a guy running a 35 billion parameter model on gtx 1060 (6 gigs of vram) at 36 tokens a second with max context size, a good speed for most tasks. Point is, these days you can run fairly capable models on ancient hardware if you know what you're doing
Oh damn. Nice find. Hurts a little that my rig is barely above his floor tho. lol.
Okay, that makes sense. The only thing I do with my GPU is play video games. So you're pretty much good to just fill VRAM pretty much all the way, depending on settings.
I found a YouTube link in your comment. Here are links to the same video on alternative frontends that protect your privacy:
Nice, I recently got this thing running on an igpu at 20 tok/sec token generation, it's surprisingly capable. Seems to be the best MoE model of it's size. I'll soon be getting a gpu server (dual rtx 5060ti 16gig), so I'm excited to see how fast it can go
its an extremely capable model at this size. i think most of the effort must be spent on miniaturizing models, as the smaller they get the more useful they become