Discuss PieFed project direction, provide feedback, ask questions, suggest improvements, and engage in conversations related to the platform organization, policies, features, and community dynamics.
How many votes do you think you do per day? Is it about 100?
edit: I looked at your profile more closely - yes, you DO vote a lot. That would explain it - the cache of recent votes, which the API uses, was too small for your volume. I've increased it by 10x, that should do.
Is there really an impact on performance/storage for using cache instead of live data for votes?
Yes, the votes table is absolutely massive (10s of GB) so even with perfect indexes, queries & joins on it are slow enough to become a problem if you are doing many things. For example when listing a page of posts in a community when there are 70 posts per page that'd mean lots of interaction with the votes table, that I'd rather avoid.
Instead, we get the last 3000 votes the viewer cast (a single DB query, or less if it's in redis already, which is another layer of cache) and hold that in memory. While creating the list of posts a "viewer has voted for this" flag is set if the post id is in the list of votes from earlier. The list of votes is sorted so an efficient binary search can be done.
That's just one part of it, there are about 3 or 4 other similar optimizations where front-loading some data drastically reduces the work involved in getting a lot of posts at once. I've put days into speeding up the 'get a list of posts' API endpoint and it's become acceptable but still not great.
The next version of the API is going to be a lot slimmer both in the responses it generates and the CPU work involved in making them. At present we are trying to produce API responses that are very close to what Lemmy provides and that brings a lot of baggage.
Thanks for the detailed explanation. I didn't expect a simple flag to have such impact, now I know.
Thanks again for creating PieFed and the continual improvements. That really is a tremendous amount of hard work.
So, my votes are gone again. It looks to have erased everything older than 14 days ago (I don't know if this is a set date [20260212] or floating 14 day hold duration).
I can't tell from your comment if this is going to keep happening (once I reach 3000 votes) or not. Can you provide some insight? Thanks.
Thank you :)
I wouldn't have guessed that much, but I'd say that's reasonable with comments added in.
@sunshine@piefed.ca - I think this is the issue you were running into with #1432. Once piefed.ca updates to incorporate the fix from @rimu@piefed.social, let us know if you still see the issue.