3211

Lemmy.world status update 2023-07-05 (lemmy.world)

submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

365 comments fedilink hide all child comments

Another day, another update.

More troubleshooting was done today. What did we do:

Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
@cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
We started using this image, and saw a big drop in CPU usage and disk load.
We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

top 50 comments

sorted by: hot top controversial new old

[-] phiresky@lemmy.world 882 points 1 year ago* (last edited 1 year ago)

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done 😁

[-] woelkchen@lemmy.world 143 points 1 year ago

Upvote causes an endless spinner on Liftoff. 😁

[-] TurnItOff_OnAgain@lemmy.world 43 points 1 year ago

I'm getting 504 gateway time outs when I try to upvote

load more comments (2 replies)

load more comments (1 replies)

[-] marsokod@lemmy.world 114 points 1 year ago

I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?

load more comments (6 replies)

[-] Snipe_AT@lemmy.atay.dev 57 points 1 year ago

aye aye sir, to the upvote machine!

[-] PatFussy@lemm.ee 27 points 1 year ago

Double the image upload size and you will see more shitposts

load more comments (1 replies)

load more comments (5 replies)

[-] Rootiest@lemmy.world 424 points 1 year ago

Test:

Upvote if you can see this comment. 👍

load more comments (10 replies)

[-] isaachernandez@lemmy.world 255 points 1 year ago

The change is noticeable. Good job guys.

Thanks for the updates.

[-] Carnelian@lemmy.world 95 points 1 year ago

I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing

[-] jake_eric@lemmy.world 23 points 1 year ago

Yup I can even post comments first try, without getting an error! Things are working well!

load more comments (1 replies)

[-] GnothiSeauton@lemmy.world 121 points 1 year ago

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

[-] AlmightySnoo@lemmy.world 58 points 1 year ago

This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.

load more comments (1 replies)

load more comments (3 replies)

[-] mintiefresh@lemmy.world 114 points 1 year ago

Wow. So much smoother today.

Great work.

You dropped this 👑

load more comments (2 replies)

[-] kionay@lemmy.world 106 points 1 year ago

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

[-] wilberfan@lemm.ee 99 points 1 year ago

I love the smell of updates in the morning.

[-] sv1sjp@lemmy.world 90 points 1 year ago

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

load more comments (6 replies)

[-] Spectator@lemmy.world 77 points 1 year ago

I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

[-] Marxine@lemmy.world 77 points 1 year ago

Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

[-] sirnak@lemmy.world 76 points 1 year ago

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

[-] cley_faye@lemmy.world 43 points 1 year ago

You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.

load more comments (1 replies)

[-] ABC123itsEASY@lemmy.world 23 points 1 year ago

Impressive if true!

load more comments (2 replies)

[-] Timou@lemmy.world 69 points 1 year ago

So that's why it was so smooth today... Great work!

[-] erik1984@lemmy.world 25 points 1 year ago

Certainly feels fast today!

[-] ericjmorey@lemmy.world 64 points 1 year ago* (last edited 1 year ago)

Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

[-] dreadedsemi@lemmy.world 61 points 1 year ago

My upvote can go through fast now

Good work

[-] xandertron@lemmy.world 60 points 1 year ago

upvoting posts is so much more stable now, we might actually see more bean posts as a result

[-] MindfuckRocketship@lemmy.world 57 points 1 year ago* (last edited 1 year ago)

Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.

That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.

Edit: This now appears resolved minutes later. All smooth on my end.

[-] ruud@lemmy.world 93 points 1 year ago

Hmm. Seems to work for me.. (Yes this is a test reply)

[-] G_Wash1776@lemmy.world 31 points 1 year ago

Everyone it’s a test reply, deploy the upvotes

load more comments (2 replies)

load more comments (3 replies)

[-] Molecular0079@lemmy.world 53 points 1 year ago

This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @ruud@lemmy.world and the rest of the lemmy.world team! The site already feels much more responsive.

[-] teraflopsweat@lemmy.world 50 points 1 year ago

Literally a night and day difference in performance and stability! Thank you all for the hard work. To other users like me, consider reducing or replacing one of your lesser used subscriptions and directing that money to Lemmy. It’s much better served here if you ask me.

[-] MrPoopyButthole@lemmy.world 49 points 1 year ago

The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.

[-] BitOneZero@lemmy.world 40 points 1 year ago* (last edited 1 year ago)

Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: !lemmyperformance@lemmy.ml

[-] _Rho_@lemmy.world 38 points 1 year ago* (last edited 11 months ago)

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like !lemmyperformance@lemmy.ml is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

load more comments (5 replies)

[-] 0235@lemmy.world 37 points 1 year ago

Appreciate that these updates use the yyyy-mm-dd format :D

load more comments (4 replies)

[-] asamson23@lemmy.world 36 points 1 year ago

It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.

load more comments (1 replies)

[-] MR_GABARISE@lemmy.world 34 points 1 year ago

This is better optimization than most enterprise devs will see in their lifetimes.

load more comments (3 replies)

[-] JoeKrogan@lemmy.world 34 points 1 year ago

Thanks to all involved across the board. Great work all around 👏👏

[-] radfordhound@programming.dev 33 points 1 year ago

It's so smooth now; the speed difference is insane! You all are doing excellent work!

[-] helios@social.ggbox.fr 32 points 1 year ago

Thanks for this very nice report.

[-] simple@lemm.ee 31 points 1 year ago

Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!

[-] xavier666@lemm.ee 31 points 1 year ago

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

load more comments (3 replies)

[-] yonerboner@lemmy.world 29 points 1 year ago

Thanks for the detailed update and all the hard work you guys are doing!

[-] ekZepp@lemmy.world 27 points 1 year ago

[-] nostalgicgamerz@lemmy.world 26 points 1 year ago* (last edited 1 year ago)

Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastadon has already said they have no interest in working with them in any capacity.

Please tell me we are defederating from Meta....or let us know what to expect

load more comments (6 replies)

[-] Jaxia@toast.ooo 25 points 1 year ago

I don't understand anything other than you worked diligently to make things smoother. Thanks to everyone for their wonderful work!

load more comments (1 replies)

[-] Oxff@lemmy.world 25 points 1 year ago

The site is running so much better now, thanks to all. BTW: Love these updates!

[-] Marxine@lemmy.world 24 points 1 year ago

Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.

Jerboa feels so damn FRESH to use now!

load more comments (2 replies)

[-] solidgrue@lemmy.world 24 points 1 year ago

Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.

load more comments