Selfhosted

60409 readers

253 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

113

Do you host your own AI? (aussie.zone)

submitted 1 week ago by SuspiciousCarrot78@aussie.zone to c/selfhosted@lemmy.world

199 comments fedilink hide all child comments

Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

you are viewing a single comment's thread
view the rest of the comments

[–] atzanteol@sh.itjust.works 1 points 5 days ago* (last edited 5 days ago) (1 children)

Yeah - I've been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it's still quite... Meh. I've been using llama.cpp and I've tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I'm finding I'm fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league...

[–] theterrasque@infosec.pub 1 points 4 days ago

When you run it, do you use unsloth's recommended settings for coding?

https://unsloth.ai/docs/models/qwen3.6

Also have preserve thinking on, it helps it stay consistent in multi turn work.

Which model version you're using can also affect results, usually unsloth's ones are good.

With all that said, it's of course a small model so it's not a super coder. The 27b is better (I'd guess 25-35% better), but of course still a small model so..

So it'll maybe not be good enough still, but should give it the chance to let it do the best it can :)