this post was submitted on 31 Mar 2026
269 points (99.6% liked)
Technology
84662 readers
3864 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I drafted several local-only designs for testing these models and that estimate looks about right (for sharding only, where most ancillary strategies come with some pretty big tradeoffs).
It turns out the actual answer in practice is specialized private compute rental, meaning the actual hardware is generally shared. That made a lot more sense to me once I had an intuitive grasp of the fact that every second of idle time on a platform capable of running these bigger models well remains an intolerable expense, simply because they are built for specialized enterprise data center infrastructure.
Put another way, to host locally you would end up needing to rent the idle time in order to make the infrastructural investment make sense financially, at which point you’re running your own data center and arrive at essentially the same destination.