Since this topic is likely to get controversial I’ll preface with my own opinion that AI CEOs are lying scumbags who grow their pockets using fear of missing out… unfortunately most other people who get hooked are other CEOs and so everyone is affected.
I don’t think current AI is anywhere close to AGI and I don’t think throwing more money at the problem will get us there. But I do think it’s a worthy question to figure out how to integrate AGI into a society with minimal friction. And by minimal I mean avoid societal collapse.
Couple of definitions since they are often ambiguous. What I imagine AGI to be is a “human-in-a-box” level of intelligence but without any basic wants of its own. You can train it “you want to help me” and it will or you can train it “you want to wage war” and it will.
The main difference between AGI and modern LLMs is the ability to learn incrementally - LLMs get trained, released, and you talk to a library in a frozen state. True AGI will integrate everything said into its model, so once you tell it “don’t output em dashes”.. it will apply that to all future conversations.
Of course, if whoever invents the AGI has the mindset of tech companies CEOs they will just outcompete all intellectual labor from everyone. Eventually the economy will reach some steady-state where no one except the company has enough money… at which point I guess there will be some sort of adjustment.
On the other hand let’s assume whoever starts the AGI is not out to stuff their pockets. The path of integration is still complex and people will get hurt one way or the other. But what’s a way to minimize that hurt? My main worry, for example, is having open-source AGI and someone like a script kiddie going “find a vulnerability in X” and it does. If a single corporation does not control the AGI then I’m sure there will be 3 000 flavors of models that can do anything. Once the integration is complete then hopefully there won’t be too many vulnerabilities to find but until then the possibility space is big. Same pattern can be observed in virtually all other aspects of our society. How to minimize this?
I bet there is no way to construct pure intelligence. Once you embed something into the real world, it will result in complex interactions. It won't necessarily be biological wants and needs. But I mean it needs power. And it better have some desire to "think" and learn. Or it might as well just sit there and do nothing all day?!
People like Asimov suggested it'd need to factor in things like not to nonchalantly harm humans. And lastly AI as in reinforcement learning foots on some goal and a score function being present.
(Edit: Also not sure how this relates to the philosophical concept of "agency". There's intelligence in the sense of knowledge of facts. Like Wikipedia is "intelligent". But we want machines to do something. That requires them to pick an option what to do / how to proceed. I'm fairly sure that requires judgement and goals.)
I think “pure” intelligence by itself is an algorithm. And as such an algorithm has no desire to do anything. Just like a web server has no desire to output web pages - it just does because it’s what the logic happens to make. It’s possible we need some sort of a quantum computer to achieve brain-like performance of intelligence - but I think intelligence is an algorithm specifically because LLMs exist.
Being able to pick an option is not necessarily a hallmark of cognition either - again a web server does that all the time. So I have a desire to think because it will end up getting me food and shelter and all the things I want. If we give a computer - a capability to think - it would have no desire to do so unless we executed the thinking algorithm.
Well, a webserver does not choose what it's going to serve. I as the intelligent entity made that decision and put it in. The webserver executes it, but there's no intelligence there. That's all mine.
I think we're pretty much at the distinction between classic computing and AI (minus things like expert systems). With traditional algorithms, the programmer puts in all the smarts. Decides what to do, how to tackle a problem, down to the mathematical operation. If X happens, then pull up some data and do Y. In case any Z happens, fail that way. All the intelligence in the process came from me, the programmer. And I made the judgements. With AI, we do not want that. We just put in the goal, and let -for example- reinforcement learning figure out what to do, and how to do it.
And with strong AI, we want even more than that. It's supposed to do a broad range of tasks, including new ones. Main point being, it should be able to figure it out. Not borrow intelligence from a human who is then going to come in to change the webserver config or reprogram the KUKA industrial robot for a new car part.
I think what you're describing is something more a traditional computer. Or an expert system. It has a lot of knowledge/data, but not any capacity to grow, learn, adapt, or figure out stuff on it's own. We need to put that in beforehand, in detail. And we already have that. It's how regular computers work.