3213

Lemmy.world status update 2023-07-05 (lemmy.world)

submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

365 comments fedilink hide all child comments

Another day, another update.

More troubleshooting was done today. What did we do:

Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
@cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
We started using this image, and saw a big drop in CPU usage and disk load.
We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

(page 7) 44 comments

sorted by: hot top controversial new old

[-] karthan@lemmy.world 1 points 1 year ago

Glorious, Ruud!

[-] zikk_transport2@lemmy.world 1 points 1 year ago

Can you describe your setup? Single instance or multi-node setup behind load balancer?

[-] ruud@lemmy.world 1 points 1 year ago

1 big server with nginx load balancing to the docker containers running Lemmy

[-] Tolstoy@lemmy.world 1 points 1 year ago

Huge thank you for all your hard work^^

[-] mintiefresh@lemmy.ca 1 points 1 year ago

Awesome. Great work all!

[-] BendyLemmy@lemmy.world 1 points 1 year ago

ROFL Back in the day, when AOL was a thing - I used to think it stood for 'Assholes Online'.

Good luck (",) appreciate the efforts you make fighting the crazy deluge.

[-] IowaMan@lemmy.world 1 points 1 year ago

Building history one update at a time 😎

[-] porksandwich9113@lemmy.world 1 points 1 year ago

Sounds like we need to comment more to me. Great work all around!

[-] Potato_in_my_anus@lemmy.ml 1 points 1 year ago

Loading is faster, thank you guys 👍

[-] ebits21@lemmy.world 1 points 1 year ago

Great work!

[-] grumpyrico@lemmy.world 1 points 1 year ago

huge improvement! respect ... massive difference to before - well done

didn't get an error so far

[-] AMillionMonkeys@lemmy.world 1 points 1 year ago

Everything is feeling great so far. The only bug I'm encountering is that when opening a thread (in Firefox on desktop) it auto-scrolls down past the content to the replies.

[-] Reverendender@lemmy.world 1 points 1 year ago

I tried to enable push notifications and the app crashed. Also, how do I create a community?

[-] daniskarma@lemmy.world 1 points 1 year ago

Amazing guys. I did donate so you can keep up with things. I really want Lemmy to last.

[-] AlmightySnoo@lemmy.world 1 points 1 year ago

I think some of the empty page issues are back now?

[-] cheerjoy@lemmy.world 1 points 1 year ago

I love how open you guys are when it comes to running the server! Much nicer than lemmy.ml imo

load more comments (1 replies)

[-] chairman@lemmy.world 1 points 1 year ago

You guys are doing god's work. Thank you thank you thank you...

[-] ghariksforge@lemmy.world 1 points 1 year ago

You guys are amazing!

[-] Rooki@lemmy.world 1 points 1 year ago

The 502 error still seems to be common by me. Vut they are less permanent. Before it stayed multiple refreshes now it is safe to say after 1 reload it is most of the time gone.