Blahaj Lemmy Meta

2877 readers

71 users here now

Blåhaj Lemmy is a Lemmy instance attached to blahaj.zone. This is a group for questions or discussions relevant to either instance.

founded 3 years ago

MODERATORS

blahaj@lemmy.blahaj.zone

supakaity@lemmy.blahaj.zone

ada@lemmy.blahaj.zone

ada@piefed.blahaj.zone

WillStealYourUsername@piefed.blahaj.zone

will_steal_your_username@lemmy.blahaj.zone

Now Fixed - Blahaj.zone latency across all services (lemmy.blahaj.zone)

submitted 3 days ago* (last edited 2 days ago) by ada@lemmy.blahaj.zone to c/main@lemmy.blahaj.zone

9 comments fedilink hide all child comments

Edit - We're back!

~~We've had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.~~

~~As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it's got plenty of space to spare, but it's not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that's happening, all of our services are running slow.~~

~~The good news is, we'll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!~~

top 9 comments

sorted by: hot top controversial new old

[–] als@lemmy.blahaj.zone 16 points 2 days ago

There's a ko-fi page if anyone wants to help support our generous overlords

[–] Nikki@lemmy.blahaj.zone 7 points 2 days ago

yay thanks

[–] rockSlayer@lemmy.blahaj.zone 6 points 2 days ago (1 children)

So glad to be back on my home instance! I noticed while I was on one of my alts that the comment I made right before the outage had a different amount of votes. Are we slow rolling sync with other servers, or is this just normal behavior?

[–] ada@lemmy.blahaj.zone 2 points 2 days ago

Are we slow rolling sync with other servers, or is this just normal behavior?

Both! When we were down, content didn't federate to us. The servers trying to send that content to us will then retry and send it again, and each time it fails, they wait longer until they try again the next time. As a result, the stuff we missed will start rolling in at different times, depending on which instance it's coming from.

But also, it's normal behaviour for vote counts to differ, because we ignore downvotes, most other instances don't. We have defederate some instances, which other instances may not have (or vice versa) and as a result, we will show different vote totals because we don't have the same voting sources.

[–] Smorty@lemmy.blahaj.zone 4 points 2 days ago

i was so sad not having lemmy ohno >o<

but now its back and thats good...

[–] GalacticSushi@lemmy.blahaj.zone 11 points 3 days ago (1 children)

Appreciate the update! I think it would be helpful to have a designated space outside of blahaj.zone where the admins can provide info in these types of scenarios. I was getting a little anxious with the instance down all day and seemingly no way for the admins to communicate what was going on.

[–] ada@lemmy.blahaj.zone 10 points 2 days ago

Normally, if it were a complete outage, we'd simply have stuck an outage notice up on the domains. In this case, it wasn't an outage initially, but just a really laggy database, so we left the services up. Once we realised that wasn't going to work, and we had to pull services down to fix the DB issue, we put up announcement pages on most of the services

[–] thoughtfuldragon@lemmy.blahaj.zone 4 points 2 days ago

I know fast storage is at a premium right now, is that something that the community can contribute to? I know very little about hosting these services so IDK if that's helpful.

[–] Cevilia@lemmy.blahaj.zone 2 points 2 days ago

Thank you so much for the update and for the hard work!