this post was submitted on 09 Jan 2025
25 points (96.3% liked)
Technology
61102 readers
5238 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Stack Overflow has been toxic for a long time already. It's one of the things that a lot of people seem pleased to see AI devour.
I've read it is still well valued because people will keep asking questions there when LLM can't answer, so they remain a precious source of post LLM curated Q&A.
...for now.
Until it's just AIs answering questions asked by other AIs while human admins block human accounts..
Valuable information source becomes bot recycling machine..? Hey, I Reddit before!
Yeah, AI has become good enough at this point that you can provide it with a large blob of context material - such as API documentation, source code, etc. - and then have it come up with its own questions and answers about it to create a corpus of "synthetic data" to train on. And you can fine-tune the synthetic data to fit the format and style that you want, such as telling it not to be snarky or passive-aggressive or whatever.
It can reproduce an api. Can't solve actual problems. LLMs are completely incapable of innovation.
That's not really true though.. They come up with brand new sentences all the time.
No, they can only take from things in their models.
Moreover, all of them use statistics, typically Bayesian, to get the results. What you get from an LLM is essentially an average* of the model data. This is why feeding LLM output into a model is so toxic, it's already the average.
They only take from the statistical distributions of words in the context of preceding words (which is why they never say "the the" etc, why the grammar is nearly always correct). But that doesn't mean that whole sentences are lifted from the source material. There are near infinite paths through those word distributions, and many have never been produced by humans, so LLMs do produce sentences that have never been uttered before.
They couldn't produce new conceptual context spaces in the way that humans can sometimes, but they can produce new combinations within existing context spaces.
Except they are. You ask about Discworld characters and it gives you direct full quotes from the books.
You realise "LLMs can quote verbatim" is not a contractdiction of "LLMs can create brand new sentences", right?