Part of what makes Lemmy (and other voting link aggregators) work is the voting aspect. By taking away outbound vote federation, it forces further consolidation into these popular instances. Thereby further exacerbate the problem because now they’re even more consolidated and the posts and comments eventually becomes the bottleneck for the exact same underlying chatty protocol. For this reason, I’d be vehemently against this change without a pairing PR that allows this information to be requested via a batch pull that the protocol makes available.
Thanks for doing all this.
Do we have any real numbers from a real server? How many votes are trying to be federated to how many servers?
Just ballparking some approximate numbers:
- !technology@lemmy.world
- 15k subscribers
- 4000 subscribed servers
- 10 votes per subscriber per day
15000 * 4000 * 10 = 600,000,000 federated actions. That is around 7,000 per second 24/7 for one community.
IMO, this real time federation just doesn't scale. We need to start planning the specs for federation batching.
Somewhat related, but why are we federating votes? Why not just federate the upvote count and downvote count? Does each server need to track the identity of every voter on a subscribed community?
Each server will track votes from their own users, preventing duplicate votes.
Why not just federate the upvote count and downvote count?
I think the answer to that is that it isn't an optimized design.
Does each server need to track the identity of every voter on a subscribed community?
I think so. Which isn't a terrible assumption that user who votes will eventually comment/post and that profile will be of use.
prototype pull
pull request for prototype: https://github.com/LemmyNet/lemmy/pull/3475
The environment variable LEMMY_SKIP_FEDERATE_VOTES is as good a way as any to reference this code hack.
Lemmy Server Performance
Lemmy Server Performance
lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.