I've been running a headless Ubuntu server for about 10 years or so. At first, it was just a file/print server, so I bought a super low power motherboard/processor to cut down on the energy bill. It's a passively cooled Intel Celeron J3455 "maxed out" with 16BG of RAM.
Since then it's ballooned into a Plex/Shinobi/Photoprism/Samba/Frigate/MQTT/Matrix/Piwigo monster. It has six drives in RAID6 and a 7th for system storage (three of the drives are through a PCI card). I'm planning on moving my server closet, and I'll be upgrading the case into a rack-mount style case. While I'm at it, I figured I could upgrade the hardware as well. I was curious what I should look for in hardware.
I've built a number of gaming PCs in the past, but I've never looked at server hardware. What features should I look for? Also, is there anything specific (besides a general purpose video card) that I can buy to speed up video encoding? It'd be nice to be able to real-time transcode video with Plex.
I think about this a lot and it really does depend on your needs.
Home lab vs home server. I like to keep them separate just because I consider my lab unstable and my home server stable. You don't have to do it this way it's just the way I like it.
https://a.co/d/6k6QpOD If you want to build a low power NAS I suggest investing in an Intel n100 based itx Nas motherboard. You can then use a case like this from Jonsbo https://a.co/d/1ayqwJV. This could be a nice cool and quiet solution. If you want to do video transcoding, the n100 has quicksync on board and with something like Truenas it's pretty easy to set up via the app catalog (check out truecharts).
If you want something even more simple (good for home users or like a backup target you keep elsewhere) I've been meaning to grab one of these "Topton 2-Bay NAS R1 PRO 12th Gen Intel N100 Network Attached Storage Media Server" from AliExpress for just this case.
As for a Lab, I suggest finding a W680 chipset based motherboard like an ASRock IMB-X1314 LGA 1700 Intel W680. You can get a cpu like a 12400 or 12500T (lower power and less heat) used cheap and you have the option to upgrade and use ecc memory without a XEON. You also have a lot more pci express connectivity.
What ever you do choose, anything pre 12th gen Intel is basically ewaste (those 11th gen mobile erying I9 engineering samples are very good but less reliable than desired). Do not invest in any old x99 based gear (unless you get it for free). I have an old dual XEON system that is still running and it uses power like a small fridge.