257

[Resolved] Federation issues with lemmy.ml (lemmy.world)

submitted 1 month ago* (last edited 3 weeks ago) by lwadmin@lemmy.world to c/lemmyworld@lemmy.world

110 comments fedilink hide all child comments

We're aware of ongoing federation issues for activities being sent to us by lemmy.ml.

We're currently working on the issue, but we don't have an ETA right now.

Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don't seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.

~~Right now we're about 1.25 days behind lemmy.ml.~~

You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.

We'll update this post when there is any new information available.

Update 2024-11-19 17:19 UTC:

~~Federation is resumed and we're down to less than 5 hours lag, the remainder should be caught up soon.~~

The root cause is still not identified unfortunately.

Update 2024-11-23 00:24 UTC:

We've explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven't really made any progress other than ruling out various potential issues.

We're currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren't able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.

As of this update we're about 2.37 days behind lemmy.ml.

We are still not aware of similar issues on other instances.

Update 2024-11-25 12:29 UTC:

We have identified the underlying issue, where a backport for a bugfix resulting in crashes in certain circumstances was accidentally reverted when another backport was applied. We have applied this patch again and we're receiving activities from lemmy.ml again. It may take an hour or so to catch up, but this time we should reliably be getting there again. We're currently 4.77 days behind.

We still don't have an explanation why the logs were missing in HAproxy after going through Cloudflare, but this shouldn't cause any further federation issues.

Update 2024-11-25 14:31 UTC:

Federation has fully caught up again.

top 50 comments

sorted by: hot top controversial new old

[-] MrKaplan@lemmy.world 70 points 1 month ago

this comment section is not a place to rant about other instances

[-] finitebanjo@lemmy.world 31 points 4 weeks ago

I don't see why not.

[-] Roflmasterbigpimp@lemmy.world 12 points 4 weeks ago

Yeah a swept clean comment section looks way better....

[-] Psythik@lemmy.world 6 points 3 weeks ago

Please defederate from them.

load more comments (10 replies)

[-] EndlessApollo@lemmy.world 6 points 3 weeks ago* (last edited 3 weeks ago)

I'll just post a sentence instead of a rant then, defederate from .ml please :3

load more comments (24 replies)

[-] greencactus@lemmy.world 50 points 1 month ago

Ah, I just today in the morning blocked Lemmy.ml. Seems to have been interesting timing :)

[-] Chozo@fedia.io 53 points 1 month ago

So it's your fault!

[-] Freefall@lemmy.world 38 points 1 month ago

In which case, thank you.

[-] credo@lemmy.world 18 points 1 month ago

No one said it was a global variable. You can hardly blame the user for poor documentation.

[-] DarkCloud@lemmy.world 23 points 1 month ago* (last edited 1 month ago)

I've never had a positive interaction with Lemmy.ml. For me it serves as a quarantine space, and a set of pre-tagged users I don't personally enjoy dealing with.

...and I'm not particularly averse to Marxists sentiments either, but they're certainly not good sales people, diplomats, or representative of their cause.

Which is just part of their reputation now. Having a bad experience with a .ml user seems to be part of the lemmy experience. It's kind of comical how consistent it seems.

That said, I'm sure there's good people on .ml.

load more comments (1 replies)

load more comments (7 replies)

[-] Lost_My_Mind@lemmy.world 37 points 3 weeks ago

I mean.....if you wanted to defederare from lemmy.ml I'd be fine with that.

[-] taipan@lemmy.world 9 points 3 weeks ago* (last edited 3 weeks ago)

It's certainly an efficient way to resolve the problem.

load more comments (8 replies)

[-] uservoid1@lemmy.world 28 points 1 month ago

Could it be an issue/compatibility with lemmy.ml running Lemmy v0.19.7 ?

[-] MrKaplan@lemmy.world 24 points 1 month ago

I don't believe it is.

There weren't any network related changes from 0.19.6 to 0.19.7 and we haven't seen this behavior with any of the 0.19.6 instances yet.

The requests are visible with details (domain, path, headers) in Cloudflare, but they're not showing on our proxy server logs at all.

[-] uranibaba@lemmy.world 29 points 1 month ago

I've read enough posts over at /r/sysadmin, it is always DNS.

[-] scsi@lemmy.world 42 points 1 month ago

[-] finitebanjo@lemmy.world 26 points 4 weeks ago

I can think of a solution.

[-] FrostyTrichs@walledgarden.xyz 19 points 4 weeks ago

FWIW the communities on walledgarden.xyz have been having federation issues to lemmy.ml for a few days as well.

Since we were/are working through some things with our host I didn't want to bother anyone from lemmy.ml about it, but it's a thing. AFAIK federation is otherwise working normally.

Good luck getting it figured out and resolved!

[-] trespasser69@lemmy.world 17 points 3 weeks ago* (last edited 3 weeks ago)

Oh, come on. Again?

The Usual suspects, Network troubleshooting editon

Source: https://lemmy.world/post/22166289

[-] Olgratin_Magmatoe@lemmy.world 7 points 3 weeks ago

FUCK active directory

load more comments (3 replies)

[-] Deestan@lemmy.world 11 points 1 month ago

Do these things usually happen from time to time?

I've noticed some lemmy.ml communities looking surprisingly "dead" some days here and there but not thought much of it.

[-] MrKaplan@lemmy.world 9 points 1 month ago

I wouldn't say usually, but they can happen from time to time for a variety of reasons.

It can be caused by overly aggressive WAF (web application firewall) configurations, proxy server misconfigurations, bugs in Lemmy and probably some more.

Proxy server misconfiguration is a common one we've seen other instances have issues with from time to time, especially when it works between Lemmy instances but e.g. Mastodon -> Lemmy not working properly, as the proxy configuration would only be specifically matching Lemmys behavior rather than spec-compliant requests.

Overly aggressive WAF configurations tend to usually being a result of instances being attacked/overloaded either by DDoS or aggressive AI service crawlers.

Usually, when there are no configuration changes on either side, issues like this don't just show up randomly.

In this case, while there was a change on the lemmy.ml side and we don't believe a change on our side fell into the time this started happening (we don't have the exact date for when the underlying issue started happening), while the behavior on the sending side might have changed with the Lemmy update, and other instances might just randomly not be affected. We currently believe that this is likely just exposing an issue on our end that already existed prior to changes on lemmy.ml, except the specific logic was previously not used.