Just saying "built on AI" or whatever isn't a convincing sales pitch. What can I actually do with AI that will improve my day to day life? Not a single advert or pitch has told me a single use case for this that applies to what anyone would use for a personal computer, and they're too risky to buy for employees in a work environment unless you can afford to be the guinea pig for this unproven line of hardware (in the sense that I know a ThinkPad will last 10 years but I have no idea how long a copilot pc will last, how often I need to replace the battery or ram or anything else). I'm aware of tech, I know what these laptops are, but as far as I can see the market for them just does not exist and I don't understand why anyone would think otherwise.
Futurology
This is such a hilarious bit of branding nonsense. There is no such thing as "AI PCs".
I mean, I technically own one, in that the branding says I do and it has a Copilot button, but... well, that's definitely not why I purchased it and I don't think I've used an "AI feature" on it. I'm not even opinionated against them, I have run local LLMs in my other computers, it's just not a good application for the device I own that is specifically branded for "AI".
The stupidity of it is that my "AI device" is an ARM device, and I absolutely love the things ARM Windows does that are actually useful. I pulled up my old x64 device that I used before I got this and man, the speed of Windows Hello, how much better it handles video streams, the efficiency... I'd never go back for a portable device at this point.
But the marketing says it's "AI", so once people start telling each other that "AI PCs" are bad and new AMD and Intel "AI" CPUs are released it's anybody's guess how the actually useful newer Windows ARM devices will fare.
I'm still hoping that the somewhat irrational anger towards "AI" stuff subsides so we can start talking about real features now, because man, this has been a frustrating generation to parse for portable Windows devices, and we still have Android, iOS and Mac devices coming down the pipe with similar branding nonsense.
I think this is referring to machines that come with APUs that have enough tensor (or whatever the equivalent is) cores in order to provide a certain baseline of "AI" operations per second, so Microsoft et al can rely on that for Recall etc without users thinking it slow.
Amusingly, the first generation of these didn't really have enough, so they're truly unwanted.
Yeah, my problem is that this was made to coincide with the Snapdragon Windows PCs, which are really good at a bunch of stuff and specifically not good at NPU performance, so the result of the "AI" branding ends up being really disappointing.
We could talk about all the other growing pains and the ways those devices were covered, but the obsessive focus on "AI" certainly didn't help, as demonstrated by the bizarre reporting linked in the OP.
What the hell do I need a laptop with an NPU for? I can write my own emails and documents. I don't give a crap about the paint and notepad AI additions. I seriously cant think of any use case for copilot plus.
What is an NPU? Nonsense Processing Unit?
Neural, I think?
It is now!
I expect that a decent amount of sales of these are just people who don't care replacing their PC.
I'd be shocked if there's in any way a statistically significant amount of these purchases driven by the "ai features".
Don't forget corporations that lease new computers every few years.
And it's hard to tell what the difference is. Apples 'built from the ground up for AI' chips just have more RAM. What's the difference with CPUs? Do they just have more onboard graphics processing that can also be used for matrix multiplication?
The stupid difference is supposed to be that they have some tensor math accelerators like the ones that have been on GPUs for three generations now. Except they're small and slow and can barely run anything locally, so if you care about "AI" you're probably using a dedicated GPU instead of a "NPU".
And because local AI features have been largely useless, so far there is no software that will, say, take advantage of NPU processing for stuff like image upscaling while using the GPU tensor calculations for in-game raytracing or whatever. You're not even offloading any workload to the NPU when you're using your GPU, regardless of what you're using it for.
For Apple stuff where it's all integrated it's probably closer to what you describe, just using the integrated GPU acceleration. I think there are some specific optimizations for the kind of tensor math used in AI as opposed to graphics, but it's mostly the same thing.
Basically yes. They come with an NPU (Neural processing unit) which is hardware acceleration for matrix multiplications. It cannot do graphics. Slap whatever NPU into the chip, boom: AI laptop!
Matrix multiplication is also largely what graphics cards do, I wonder how the npus are different.
Modern graphics cards pack a lot of functionality. Shading units, Ray tracing, video encoding/deciding. NPU is just the part needed to accelerat Neural nets.
But you can accelerate nural nets better with a GPU, right? They've got a lot more parallel matrix multiplication compute than any npu you can slap on a CPU.
It all depends on the GPU. If it's something integrated in the CPU it will probably not so better, if it's a 2000$ dedicated GPU with 48GB of VRAM is will be very powerful for Neural Net computing. NPUs are most often implemented as small, low-power, embedded solutions. Their goal is not to compete with data centers or workstations, it's to enable some basic "AI" features on portable devices. E.g: "smart" camera with object recognition to give you alerts.
The Apple chips also have a wide interface to the RAM. That means you can run chatbots (LLMs) and other AI workloads that are memory-bound at crazy speeds compared to an Intel (or AMD) computer.
Really? How fast is the memory bus compared to x86? And did they just double the bus bandwidth by doubling the memory?
I'm dubious because they only now went to 16gb ram as base, which has been standard on x86 for almost a decade.
Depending on the chip, they have somewhere from 100 to 400 GB/s. I'm not sure on the numbers on Intel processors. I think the consumer processors have about 50 - 80 GB/s. (~Alder Lake, dual channel DDR5) Mine seems to have way less. And a recent GPU will be somewhere in the range of 400 to 1000 GB/s. But consumer graphics cards stop at 24GB of VRAM and these flagship models are super expensive. Even compared to Apple products.
The people from the llama.cpp project did some measurements and I believe the Apple "Metal" framework seems to outperform the x86 computers by an order of magnitude or so. I'm not sure, it's been some time since i skimmed the discussions on their Github page.
Apple is also much faster because the integrated graphics are actually usable for LLMs.
The base M is just a big faster than an Intel/AMD laptop if you can get their graphics working. The M Pro is 2x is fast (as its memory bus is 2x as wide). The M Max is 4x as fast.
AMD is coming out with something more competitive in 2025 though, Strix Halo.
It’s a double whammy. No one wants AI. No one likes Windows 11.
It's a no from me. I suspect as the US gets more deregulated for AI, it will be more no's from people around the world.
Have they tried forcing people to upgrade to AI PCs in order to receive security updates by checking to see if your PC is an AI PC? You know just to prove people really want AI PCs?
/s
It isn't that surprising.. they are shipping AI chips, but there is almost no software that makes use of them yet. Unless people require them for a certain use case, why bother?
Reminds me a bit of "5G ready" phone contracts, just meaningless marketing. Don't need to chase every hype.
Marketing ruining it for themselves. Good job.
”However, if it is performance you are concerned about, "it's important to note that GPUs still far outperform NPUs in terms of raw performance," Jessop said, while NPUs are more power-efficient and better suited for running perpetually.”
Ok, so if you want to run your local LLM on your desktop, use your GPU. If you’re doing that on a laptop in a cafe, get a laptop with an NPU. If you don’t care about either, you don’t need to think about these AI PCs.
Or use a laptop with a GPU? An npu seems to just be slightly upgraded onboard graphics.
It’s a power efficiency thing. According to the article, a GPU gets the job done, but uses more energy to get there. Probably not a big deal unless charging opportunities are scarce.
What kind of AI workloads are these NPUs good at? I mean it can't be most of generative AI like LLMs, since that's mainly limited by the memory bandwith and at this point it doesn't really matter if you have a NPU, GPU or CPU... You first need lots of fast RAM and a wide interface to it.
That's why NPU will have high bandwidth memory on chip. They're also low precision to save power but massively parallel. A GPU and CPU can do it too, but less optimized.
That was my question... How much on-chip memory do they have? And what are applications for that amount of memory? I think an image generator needs like 4-5GB and a LLM that's smart enough as a general porpose chatbot needs like 8-10GB. More will be better. And at that point you'd better make it unified memory like with the M-series Macs or other APUs? Or this isn't targeted at generative AI but some other applications. Hence my question.
Last I heard this is for onboard speech recognition and basic image recognition/OCR so these things can more intelligently listen, see and store what you're doing without sending it to a server. Not creepy at all.
I will block the AI on any computer I use.
Because AI still sucks. It's incredibly unuseful for day to day operations, and only useful in very specific scenarios that I, personally, only need access to it like 2-3 times a year. It's not worth having it on my personal devices when I can just go to chatgpt website or something when I want to give it a shot.
Can someone point me to technical/learning resources about NPUs? So far all I have seen is superficial marketing talk and ads. And on top of that, everything existing in the AI/ML sector still seems to require beefy server hardware. So is there any real point to NPUs at all?
I got one for a good price a few months ago to replace my dying Surface Pro 3.
No regrets, but I love it for the speed and the screen, not all the AI bullshit.
No issues no my end with ARM; it doesn't play games well but I have a gaming PC for that.
So, I might build a new machine next year. Does that mean motherboards will have an extra NPU chip on them? Or are new CPUs supposed to come out with the extra functionality?
I won't be using it in any case, as 1. I'm not running Windows, and 2. there most likely wouldn't be any purpose to it even if I was, but this will probably contribute to making hardware more expensive for no gain (yay).
The newer mobile chips have some sort of npu type silicon included. I don't think it's very widespread on desktop yet.
They are complete ass.
What's ironic is that the local llm/diffusion communities will not touch these. They're just too slow, and impossibley finicky to set up with models big enough for people to actually want.
AMD's next gen could change that, but they've already poisoned the branding. Good job.