this post was submitted on 31 Jul 2026
39 points (100.0% liked)
chat
8640 readers
183 users here now
Chat is a text only community for casual conversation, please keep shitposting to the absolute minimum. This is intended to be a separate space from c/chapotraphouse or the daily megathread. Chat does this by being a long-form community where topics will remain from day to day unlike the megathread, and it is distinct from c/chapotraphouse in that we ask you to engage in this community in a genuine way. Please keep shitposting, bits, and irony to a minimum.
As with all communities posts need to abide by the code of conduct, additionally moderators will remove any posts or comments deemed to be inappropriate.
Thank you and happy chatting!
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
if youre on this app and subscribe to a socialist worldview then i hope you can understand that when you use something for free you are the product (unless you use a socialist AI model)
i run chinese models locally. there is no way for them to make money off me. i even run my own searxng mcp-server instance with tor added. theres zero way for anything to be tracked or profited from.
I'm curious what model you run and on what hardware?
qwen 3.6 35b a3b, 2k usd custom build computer from 2-3 years ago with about 80gb cumulative vram/ram that was partially paid for by my employer. i do nondescript science™️ for my job. its pretty useful for dredging the internet for sources, especially sources that would be normally inaccessible to me because i dont speak the language the paper is in. my previous computer was falling to bits (nearly 20 years old, i hope i can make this one go that far) so i needed a good one anyways.
How on earth do you have 80 gigs of VRAM?
cumulative, i only have ~12gb vram
Oh, I see. So, I have 44gb cumulative ram, 12gb VRAM and 32gb "regular".
I was hoping there was some magical way to improve my 4070ti (not super).
this model can be compressed to as low as 12 gigs (though I wouldn't recommend going that low due to a huge quality drop). And not all of this kind of model has to reside in vram - you can have most of it in ram and the rest gets cached in vram
I'm curious, can the model fill your whole VRAM or do you have to leave extra space for the actual, you know, whatever it's doing? Like, could you actually do anything with the 12gb model if you only have 12 GB of VRAM?
Yes, you need to leave some space for the context (kv cache) - a lot of space if you need a lot of context. Large context just allows you to generate more text and run longer tasks.
12 gigs is actually a good amount to run this class of model. As I said, not all of the model has to live in the vram
here is an example of a guy running a 35 billion parameter model on gtx 1060 (6 gigs of vram) at 36 tokens a second with max context size, a good speed for most tasks. Point is, these days you can run fairly capable models on ancient hardware if you know what you're doing
Oh damn. Nice find. Hurts a little that my rig is barely above his floor tho. lol.
Okay, that makes sense. The only thing I do with my GPU is play video games. So you're pretty much good to just fill VRAM pretty much all the way, depending on settings.
I found a YouTube link in your comment. Here are links to the same video on alternative frontends that protect your privacy:
Nice, I recently got this thing running on an igpu at 20 tok/sec token generation, it's surprisingly capable. Seems to be the best MoE model of it's size. I'll soon be getting a gpu server (dual rtx 5060ti 16gig), so I'm excited to see how fast it can go
its an extremely capable model at this size. i think most of the effort must be spent on miniaturizing models, as the smaller they get the more useful they become
china isnt socialist cause they make llms
Oh that's not what I'm saying at all. I fully suppot the Alibaba Corporation and their socialist stochastic text generators
https://www.businessinsider.com/xi-jinping-china-ai-wins-new-year-address-2026-1?op=1
edit better source
https://english.www.gov.cn/news/202607/17/content_WS6a5a1172c6d00ca5f9a0c46b.html
Do you realize that socialists aren't obligated to be in favor of something just because Xi is?
he was saying ai models cannot be made in a socialist way. the leader of the largest communist party on earth endorses their open development for the automation of the economy. these two statements are incongruent if you think china is doing things correctly.
And despite how often people are brow-beaten for saying so on this website, not everyone thinks China is doing everything correctly.
LLMs are intrinsically bad because they degrade people's brains and frankly I personally think the automation of thought simply is not something that is in line with human flourishing, that doesn't change if such things are apparently happening Socialistly.
to automate many forms of work you are inherently automating away thought. most families had people whose whole job was weaving and making clothes for the family. with the advent of modern industrial scale clothing manufacturing processes, almost no one does this outside of hobbyists.
Apples to oranges - while it is true physical skills are ultimately cognitive ones, there is a very clear difference between the automation of tasks that are reliant on the mind-body connection being finely tuned and automation of basic, incredibly important shit like critical thinking, the ability to research information, and literacy.
americans werent capable of critical thinking in the first place. maybe if chinese models outcompete western ones they can bolt on critical thinking to your average american flesh bag and finally america will have communism
That's not much of an argument. Are you actually a socialist or are just an unironic campist?
Think materially: the average American is so stupid because of a combination of deliberate effort to make the population stupider, and because of conditions that have fostered a devaluation of intellectual skills.
LLMs have only made both problems worse. Why do you think things will go any differently in China? And don't just cite that they're socialist so it'd be fine, give me actual material differences that mean LLMs won't have the same negative cognitive effects they've had literally everywhere else.
i am making a joke because i dont take you seriously.
think materially: the human population is set to stagnate at around 10b and will not grow further. responsible governance means accelerating automation in order to take care of a mass of humans who will be elderly and incapable of many things.
keep moving the point away from you not valuing the intelligence involved in much of manual labor, including weaving, and the low-lying misogynistic tones of that argument...
you can argue theres cognitive effects from many thousands of things. i drink alcohol and it erodes my critical thinking too, but you don't see many luddites-masquerading-as-socialists demanding this new-fangled technology called fermentation be banned. each new relation and technology creates new ways of thinking. the industrial revolution was proof of that, but did marx say we needed to stop the presses? no, he said we needed to seize the presses for the good of society. any new technology can be used for good or bad, and just because many people misuse a technology it doesn't mean that technology wouldnt have benefits when used by a socialist group.
I'd put in the effort to fully respond to this, but if you're going to misogyny jacket me there's no particular point.
I quite directly acknowledged the fact that there is significant cognitive skill in weaving, literally my only contention was that it was not so nearly as fundamental of a cognitive skill as the things LLMs degrade. And even beyond that I actually do think that people should be learning mind-body cognitive skills like that, regardless of usefulness in modern society (and there are many that are very useful) I just didn't feel a need to make that clear because that's not what we're discussing.
But you're probably not even going to read that, given you've already resorted to misogyny jacketing the removed you're talking to because you don't have a real argument, so like, go fuck yourself.