Technology

The article talks a lot trash about AMD and ROCm but vulkan works fine too. In fact from a datacenter GPU standpoint there is an AMD option called the V620 available on US EBay that I was able to haggle to $350, with 32GB VRAM, 512GB/s bandwidth, and runs the same Qwen-3.6-27b at about 20t/s. It requires a few of the same fan shenanigans this guy did but there is no need to pull specific past software versions to make it usable in Linux

Here I was expecting a graphics demo to blow our collective minds but instead I got a story about a local LLM for cheap. It is . I should have known better.

Were I the author / tech cobbler here, I'd be concerned that too much time with an LLM, local or otherwise, might erode or dull my apparently fairly sharp reasoning and tech skills. (Clarification: Not my sharpness, theirs. I'm a potato.)

Other thoughts: For a minute I thought this whole thing was a tribute to, or a troll in the manner of, that one Redditor that always spun their stories around to being about their dad beating them with jumper cables.

Also, my old PC developed an issue like the warm reboot problem, except with the network interface. I couldn't just restart, I had to power off and back on. I never did bother to find out whether it was early signs of hardware failure or whether it was an old hardware / newer kernel mismatch.

Seems like an awful lot of trouble to save $100 not buying a 5060 Ti that also has 16GB.

Sure it's got a lot of VRAM, but the 4080 has five times the compute power.

The main thing that itches me with the V100 is the fact that given that pascal is about to be EOL, a 2017 card is probably soon next

Why tho?

82db? Wild. I guess they don't really care that much about noise in data centers though.

eBay has some rad Chinese mezzanine boards for these guys too. Nvlink works and everything lol

Yeah but the 0 point doing this unless you want to run AI models for some reason. These GPUs can't do video game graphics so this isn't a solution to the GPU shortage.

This is a bit like me writing an article about NASCAR, now I can turn left whenever I want. But I haven't magically acquired a functional vehicle for a fraction of its value. I've purchased a second hand specialist product that is usually useless outside of that environment.

Not an AI guy, but I do like using niche hardware wrong to get results cheap. Can anyone tell me what this would be like for gaming or general computing? My 1660 super was a budget pick when I got it back in '18.

This isn't the smart way, though.

What the homelabbers do (at least before the RAM crisis) is buy Xeon/TR/EPYC boards on the cheap, and then run gaming GPUs for hybrid inference.

This is what I do. I run MiMo 2.5 at 8-10t/s on a 7800X3D/RTX 3090/128GB CPU RAM, more with Dflash. That's a 300B model: it's not even in the same class as Qwen 27B, which is what the dev in OP's article is trying to run.

And this is small-time: setups with 4-8 memory channels can run stuff like Kimi or Deepseek Pro, even faster. Or they can run smaller LLMs with quantization types that are very fast on CPUs, and get crazy speeds.

...And besides, Qwen 27B can run fine on a 4080, with the right framework. It will fit in 16GB as an exl3.

Not that this isn't a cool hardware hacking project.

...But it's kind of the wrong approach. It's about 2 years out of date, as MoEs are king in LLM land now. RAM is horrendously expensive, yes, but so are most used V100s, or used 3090s.