chat

8640 readers

183 users here now

Chat is a text only community for casual conversation, please keep shitposting to the absolute minimum. This is intended to be a separate space from c/chapotraphouse or the daily megathread. Chat does this by being a long-form community where topics will remain from day to day unlike the megathread, and it is distinct from c/chapotraphouse in that we ask you to engage in this community in a genuine way. Please keep shitposting, bits, and irony to a minimum.

As with all communities posts need to abide by the code of conduct, additionally moderators will remove any posts or comments deemed to be inappropriate.

Thank you and happy chatting!

founded 5 years ago

MODERATORS

SexUnderSocialism@hexbear.net

Sulvy@hexbear.net

oscardejarjayes@hexbear.net

Wakmrow@hexbear.net

LeninWeave@hexbear.net

abc@hexbear.net

many AI users are scared of real things (hexbear.net)

submitted 3 days ago by opiumfree@hexbear.net to c/chat@hexbear.net

52 comments fedilink hide all child comments

theyre scared of doing their own art because they risk being ridiculed for mediocre tehnique. they vent to chatgpt because theyre afraid of being rejected in early friendships and just want someone to say they love them. they dont go to therapy because 9 dollars a month is way cheaper than 800 dollars. they use AI chatbots for sex because theyre afraid of the very real risks of sex and dont feel like they can get it with a real person. we used to vent online but now post pandemic you risk your shit being posted on a commentary channel blasted to millions of people in your most vulnerable times.

you are viewing a single comment's thread
view the rest of the comments

[–] kleeon@hexbear.net 4 points 2 days ago (1 children)

this model can be compressed to as low as 12 gigs (though I wouldn't recommend going that low due to a huge quality drop). And not all of this kind of model has to reside in vram - you can have most of it in ram and the rest gets cached in vram

[–] EmmaGoldman@hexbear.net 2 points 2 days ago (1 children)

I'm curious, can the model fill your whole VRAM or do you have to leave extra space for the actual, you know, whatever it's doing? Like, could you actually do anything with the 12gb model if you only have 12 GB of VRAM?

[–] kleeon@hexbear.net 4 points 2 days ago (3 children)

I'm curious, can the model fill your whole VRAM or do you have to leave extra space for the actual, you know, whatever it's doing?

Yes, you need to leave some space for the context (kv cache) - a lot of space if you need a lot of context. Large context just allows you to generate more text and run longer tasks.

Like, could you actually do anything with the 12gb model if you only have 12 GB of VRAM?

12 gigs is actually a good amount to run this class of model. As I said, not all of the model has to live in the vram

here is an example of a guy running a 35 billion parameter model on gtx 1060 (6 gigs of vram) at 36 tokens a second with max context size, a good speed for most tasks. Point is, these days you can run fairly capable models on ancient hardware if you know what you're doing

[–] JoeByeThen@hexbear.net 1 points 1 day ago

here is an example of a guy running a 35 billion parameter model on gtx 1060 (6 gigs of vram) at 36 tokens a second with max context size, a good speed for most tasks. Point is, these days you can run fairly capable models on ancient hardware if you know what you're doing

Oh damn. Nice find. Hurts a little that my rig is barely above his floor tho. lol.

[–] EmmaGoldman@hexbear.net 2 points 2 days ago* (last edited 2 days ago)

Okay, that makes sense. The only thing I do with my GPU is play video games. So you're pretty much good to just fill VRAM pretty much all the way, depending on settings.

[–] HexReplyBot@hexbear.net 1 points 2 days ago

I found a YouTube link in your comment. Here are links to the same video on alternative frontends that protect your privacy: