Hi all. I've been building my own little fediverse setup at dboft.com to fit how I like to read, and I'm grateful for all the work and community that makes something like this possible. Looking forward to being here with you all!

you are viewing a single comment's thread
view the rest of the comments

[–] kalman@dboft.com 5 points 5 days ago

Pretty much, yes. Embedding a post takes about half a second on dboft's small server, and the lookup afterwards is under a millisecond. At 3–4 thousand new posts and comments a day that's maybe 40 minutes of work, so the ceiling is exactly what you guessed: how fast new posts can be embedded. More cores or a small GPU would push it way out i would think. Storage grows too, about 14 KB per post with its index, so 35k today is ~500 MB and a million would be ~14 GB. It all lives on disk in Postgres; keeping the index in RAM just makes lookups sub-millisecond instead of a few milliseconds, and Postgres caches the busy parts on its own, so it degrades gently rather than falling over. This was just an idea; I'm just putting it into together to see if it holds up. I like the idea of a shared service... I wonder what the memory constraints would be at scale.