Ask Lemmy
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
There's plenty of open source models that don't really have any restrictions, you just have to host them yourself (which you can do on your own computer if you have a decent gpu)
for example: mixtral 8x7b
just use koboldcpp or something similar to run the GGUF files and you're good
for example:
Isn't that one also pretty censored? Really uncensored one usually either builded from scratch (behemoth or midnight-miqu as example) or named accordingly: mixtral-uncensored or llama3-ablitered.
I've never personally had issues with 8x7b refusing requests, but I guess I haven't really plumbed the depths on what it might agree or disagree to, I have run it through the ordinary gambit (pretend to be public figure x, make dangerous item y, say untrue thing about company z) and it hasn't given me any problems but sure whatever works for you
If you have a good enough NVIDIA card, probably a 1080ti or better, download KoboldCPP and a .gguf model from huggingface and run it locally.
The quality is directly tied to your GPU's vram size and how big of a model you can load into it, so don't expect the same results as an LLM running on a data center. For example, I can load a 20gb gguf model into a 3090 with 24gb of vram.
Actually not 100% true, you can offload a portion of the model into ram to save VRAM to save money on a crazy gpu and still run a decent model, it just takes a bit longer. I personally can wait a minute for a detailed answer instead of needing it in 5 seconds but of course YMMV
Is there a general term for the setting that offloads the model into RAM? I'd love to be able to load larger models.
I thought CUDA was supposed to just supposed to treat VRAM and regular RAM as one resource, but that doesn't seem to be correct.
is there a general term for the setting that offloads the model into RAM? I’d love to be able to load larger models.
Ollama does that by default, but prioritizes gpu above regular ram and cpu. In fact, it's other feature that often doesn't work, cause they can't fix the damn bug that we reported a year ago - mmap. That feature allows you to load and use model directly from disk (alto, incredibly slow, but allows to run something like deepseek that weight ~700gb with at least 1-3 token\s).
num_gpu allows you to specify how much to load into GPU vram, the rest will be swapped to regular RAM.
You're probably looking for an abliterated model. Be sure you can run it, first, as localhosting models needs high VRAM. You'll want RAM, too, in the case of GGUF models.
I'd have to write a whole half a book here to explain how to use them, but that information is freely available online. If you don't have a beefy GPU, look into how to host GGUF models.
I don't think anything is fully uncensored. As for the similar question about least biased I'd say the models from Mistral.