348
submitted 10 months ago by dessalines@lemmy.ml to c/announcements@lemmy.ml

Sometime early this morning, our database started having issues, most likely due to a disk corruption issue.

Unfortunately the postgres database was unrecoverable, and we had to restore from a nightly backup, so a few hours of posts and comments will be missing.

Apologies again for the downtime, and for the disruption.

all 43 comments
sorted by: hot top controversial new old
[-] nutomic@lemmy.ml 91 points 10 months ago
[-] Stache_@lemmy.ml 36 points 10 months ago

Oh holy cow, those servers are running at some pretty high temps!

[-] TheSaus@lemmy.dbzer0.com 20 points 10 months ago

Liquid cooling required

[-] morrowind@lemmy.ml 7 points 10 months ago

That's the power of optimized rust πŸ¦€πŸ¦€

[-] dandroid@dandroid.app 11 points 10 months ago

Holy fuck. I thought this was a joke before I saw who was posting it.

[-] bigkahuna1986@lemmy.ml 46 points 10 months ago

Do you know that you are doing to me?! I actually had to work while lemmy was down...

[-] qprimed@lemmy.ml 15 points 10 months ago

no kidding. this particular FWP was... unpleasant.

[-] Alaskaball@hexbear.net 3 points 10 months ago

Join us! Enjoy being among posters that never stop posting hexbear-retro

[-] v_krishna@lemmy.ml 33 points 10 months ago

How does lemmy federation work in this case? Conceivably after being restored from backup the lemmy.ml instance could see those few hours of lost history as federated to other lemmy instances and resync it back as the host instance. Obv I'm vastly oversimplifiying things but what happens today?

[-] dessalines@lemmy.ml 18 points 10 months ago

I believe they're lost, as recieved apub content doesn't have the original post / comments, but I'm not totally sure. cc @nutomic@lemmy.ml

[-] ada@lemmy.blahaj.zone 16 points 10 months ago

It strikes me that there is the potential to use trusted remote servers as a means of recovering the lost data. I mean, nearly every lemmy instance except lemmy.ml will have copies of the missing data, and given the hugely redundant availability of that data (including the ability to compare from multiple sources to establish/verify trust), using that data to rebuild missing content seems like it could be useful functionality.

[-] heyfrancis@lemmy.ml 3 points 10 months ago

If I understand, federating contents through ActivityPub only works once. Sounds a good feature to re-download contents again, but may introduce additional work as there should be some way to know if a content is missing then another job to rebuild

[-] nutomic@lemmy.ml 3 points 10 months ago

That doesnt work because we generally cant trust remote servers. Plus we dont even know where to fetch from, so wed have to run a complete crawl of all known instances which isnt practical.

[-] RealityCavesIn@lemmy.ml 29 points 10 months ago

Awesome that it's back up! Thanks for all your hard work

[-] dessalines@lemmy.ml 22 points 10 months ago
[-] ipkpjersi@lemmy.ml 27 points 10 months ago

No worries, glad to hear you got it up and running again :)

[-] magnetosphere@kbin.social 24 points 10 months ago

No apologies necessary. Y’all have been working overtime to get things back in order, and I appreciate it!

[-] spaphy@lemmy.ml 20 points 10 months ago

I AM TALKING IN ALL CAPS BECAUSE I AM MAD

HOW DARE THIS FREE SERVICE I USE GO DOWN HOW EVER WILL I POST MEMES FROM MY OPAL THRONE

Thanks for coming to my Ted talk.

~~also thank you for hosting us and working on the issue!~~

[-] Omega_Haxors@lemmy.ml 13 points 10 months ago* (last edited 10 months ago)

When I saw LemmyMl was down I went to sleep and had a dream where I was eating extremely large fries that that were labeled on the fryholder "Sadistically Large" (chicken fries as well as potato fries) and drinking probiotic shakes so dangerous they literally had to be bundled with a free antibiotic shake by law which of course I took one sip of in front of the clerk to calm them down before guzzling the other shake down with impunity. Woke up before I found find out what the burger was. Maybe those anti-woke types are on to something.

[-] chaorace@lemmy.sdf.org 2 points 10 months ago

This was, without any doubt, a prophetic dream. The apocalypse will soon be upon us! Be wary of the four horsemen: Ronald, King, Wendy, and... Chihuahua

[-] originalucifer@moist.catsweat.com 11 points 10 months ago

looks like some replay action too. my instance just got spanked.

[-] TiredNerdDad@lemmy.ml 8 points 10 months ago

Welcome back!

[-] Ramin_HAL9001@lemmy.ml 8 points 10 months ago* (last edited 10 months ago)

I saw the "502 Gateway" HTTP error message, but I never even closed my browser tab. I refreshed again today and Lemmy.ml was back online! Thanks for your hard work!

[-] Lettuceeatlettuce@lemmy.ml 7 points 10 months ago
[-] Zerush@lemmy.ml 12 points 10 months ago
[-] Aviandelight@mander.xyz 6 points 10 months ago

Please tell me you've been saving this picture just for this type of occasion. πŸ˜‚

[-] Zerush@lemmy.ml 4 points 10 months ago

Nope, only searched Lemmy server. πŸ€₯

[-] sabreW4K3@lemmy.tf 5 points 10 months ago

Glad to see it back

[-] bdonvr@thelemmy.club 2 points 10 months ago

Remember to make regular backups, kiddos.

[-] Pratai@lemmy.ca 1 points 10 months ago
this post was submitted on 08 Jan 2024
348 points (96.0% liked)

Announcements

19 readers
3 users here now

Official announcements from the Lemmy project. Subscribe to this community or add it to your RSS reader in order to be notified about new releases and important updates.

You can also find major news on join-lemmy.org

founded 5 years ago
MODERATORS