Mondez

I was thinking more about the cd producer and purchaser. If the purchaser knows they are getting bootlegs it's just plain old copyright infringement, if not it's counterfeiting.

Sure as long as everyone involved knows that is what is going on.

Counterfeiting is what has gone on here.

It won't kill us as a species, but its use may lead to disasters that were unintended and didn't need to happen. Much of the regulation in other industries is due to that kind of thing happening. Worse, the companies making them are trying to get regulation to curtail other parties development for economic advantage rather than regulation on use which will steer the entire industry.

It still won't destroy the planet but it will change the environment enough to screw over a large part of the population and cause massive social upheaval.

Making a choice involves understanding the things been chosen between. The only choices LLM chat bots are making is what series if tokens best fits the prompt they were given. There is no more consideration than that. Then results like this suddenly don't seem surprising and infact are the expected outcome.

Point 3 is marketing, the LLM was a tool used by humans to solve a math problem and indeed relied on a lot of human input to achieve even that. LLM connects dots humans already plotted isn't as catchy though.

Except LLMs haven't show themselves capable of that at all, they have no agency, they just generate output based on their input Ina somewhat chaotic manner.

Tools have been built that basically chain LLM output into input and given some starting directions get the LLM to generate commands that can be ran on the local machine including commands to run commands on other machines. Because various conversations on exploits and hacking and sample code are in its training corpus it will generate commands to do these things. Not because the LLM or the agent program is smart in anyway but I fact because it's dumb as bricks and has no concept of what it's doing or the consequences might be.

Ultimately a human has used it as a tool to achieve the hacking. It's might not be the intended achievement but that is on the human running a process with chaotic output with sufficient privileges with no oversight.

LLM tools are like taking a helicopter that can only drop you off at the site because somone put down the trail markers and made the maps. It can't figure out how to go anywhere using a method a human didn't already do most of the important work finding out.

They still don't get it right all the time, they just stacked a few together to filter out the obviously wrong stuff.

It's not exactly what you wrote, you only explicitly wrote it doesn't follow unwritten rules. I think we are both being pedantic over the basic premise though, LLM output dumb.

It doesn't understand or follow written rules, never mind unwritten ones.