Technology
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
The main issue with Open / Local LLMs is the lack of broad support. It's not hard for me to run Local LLMs, but apparently a decade of informal python experience and using linux mint is intimidating to some people. Apparently people want to click button and talk to phone god? Mostly speaks of a failing education system if you ask me.
I'm not even that good with AI, I just lurked a bit on !LocalLLaMA@sh.itjust.works and got mine running at like 30 tokens/second on my laptop.
I will have to defend users here and blame AI scientist/developers together with the whole docker culture. The way they package and delivery software is just terrible. And even though pytourch has support for all GPU apis the devs seems to have a hard time adding this support. Granted, part of that is also on pytourch that for some reason Don't seem to be able to make a unified release that supports all GPUs and then you need different versions and by default IIRC they default package only supprt CUDA and CPU.
Thats why I'm actually at least for now hoping for the FOSS Chinese Frontier labs to make the breakthroughs since many of them still share research and open weight models
Most of us here think streaming is useful, despite the fact that streaming services are based on the idea that they own and control all access to media with you renting it from them, raising prices, shoving ads in your face, and removing media from the collection you’re renting from them. Our solution in this community is investing in the open source community around Jellyfin, the Arr stack, etc. so we can still enjoy the benefits of streaming services while owning our own content and not having to financially support companies we don’t agree with. I’d say a lot of us here who are happily using Jellyfin might otherwise have a streaming account if it didn’t exist. Admittedly, I used Netflix until I realized I had better options.
I think the same is true with local LLMs. Not all of us agree that LLMs are useful, but most of us here agree that a few tech giants tightly controlling LLMs and renting them to everyone is not going to be a good thing. Without self-hosted LLMs, many people who do find value using them will go ahead and financially support the rent-seekers who are hell bent on destroying the world for their own financial gain, as well as support them by sharing data that can be used to train their models. Even when you use the free tier of Chat GPT, for example, you’re supporting OpenAI by giving them your prompts that they can use to make their models better.
The ecosystem around running open weight models is rapidly evolving. I’m already running the Qwen 3.6 MoE model with the desktop beta of OpenCode on my gaming laptop and it’s pretty decent. Personally, I’ve found ways to use LLMs where they are actually useful and not just slop generators, though I initially thought they were useless before I spent a lot of time working with them. I’m all for supporting and contributing to this ecosystem so that people can use LLMs without giving their money and data to shithead psychopaths.
👏 This is the way to do it!
Mmm.. better that all AI fail.
Perhaps in Candyland, or maybe even the land of chocolate. But I think in the real world it's here to stay.
yeah, we only created the garbage, how could we ever turn the garbage off?
I mean, with shit takes like that, geeze, guess we're really fucked
You know, just like actual garbage, not all AI is equal. Some ai you probably do want to throw away. Some can be repurposed into something great. locally hosted AI is a wonderful tool that empowers the individual. The same as any FOSS software, it's a threat to the establishment.
To conflate the mathematics and technology of LLMs with billionaire bullshit is tempting, but also an error.
Yeah, we should stop mislabeling LLMs as AI. I have yet to see anything, ANY THING, that would justify the artist crushing, soul sucking, component costs-skyrocketing, child undressing, suicide prompting, power guzzling, water draining and manifold other garbage that is intrinsic on this 'age of AI'.
It's snake oil, shitware and techbros burning the fucking planet and destroying entire industries because they're so fucking sure their roided up elizabots are gonna change humanity - and can't comprehend, it certainly is changing humanity, FOR THE WORSE.
It's like people are starting to realize what the "luddites" were saying from the beginning.
This tech is not from the people to the people like the web, it's from big corpos to fuck you.
You won't have that tech, you will rent a highly modified one at best, built with the purpose to manipulate you.
Remember: there is a club out there, and you are not part of it.
You must be new to the IT then.
Clouds are only US, and they hold all the data and traffic for most of the world.
Before them, operating systems - everyone and their mother using Windows.
Phones? Google or Apple.
Communication? Teams. Shared documents? Exchange. Countries are balls and hair deep in the US since the 90s and nobody really cared to do anything about it. SAP, Sellsforce, Palantir, Jira, Confluence, Github, many others I'm not even aware of, all US companies holding everything that companies and countries need in the US way before any AI. Which doesn't even fucking work.
None of this is new.
Alternatives are there, but they need the money everyone sends to the US.
I just wish we could invest the time/money/resources into compressing AI and making it smaller and more efficient. I'd so much rather have a somewhat capable AI that can be run locally and offline, to outsource menial tasks to like alphabetizing spreadsheets and so basic image modification, than to have to upgrade my hardware constantly or use cloud based SaaS and/or have newer models that are more accurate in their predictions.
Of course that assumes a lot of things, like the intent to help people and not make money. Maybe someone in the Linux-sphere will make something.
Look into Bonsai Ternary models. They're "1.5" bit models that have to be trained that way (so no taking a full model and quantizing it down) but they are so efficient and they can run on CPU only, though it's a bit alpha at the moment. Really cool company and projects.
You have to create a specific environment for them though, using Bonsai's GGUF version which enables them to run properly. So unfortunately, no use in LM Studio yet.
Did anybody click the link to read the paragraph-long website text? Not only is it packed with assumptions about AI, it's simply unhinged.
The ability to... run intelligence systems without asking permission is of existential importance.
Existential?! No it's not.
This mirrors the delusion Sam Altman demonstrated when he insisted nobody could raise a child without AI.
I don't think they mean that having AI is vital. I think they mean it's vital that the rich not have a monopoly on AI.
Regardless whether AI is useful, its production would never have become this harmful if a rich person didn't think they could own it and make money with it. Making a thing publicly accessible makes it less attractive to capitalists.
"Of existential importance" literally means they think it's vital to existence.