79

Deleted something I shouldn't have. I learned my lesson, but I had to revert to a backup that was about 3 days old. My bad.

top 23 comments
sorted by: hot top controversial new old
[-] mouse@midwest.social 17 points 1 year ago

We all make mistakes, thank you for being transparent. 💖

[-] Nemo@midwest.social 13 points 1 year ago

Hey, can you add Spanish back as a language option? That's another thing that got reset.

[-] SwingingTheLamp@midwest.social 9 points 1 year ago

Whoops. ¯\_(ツ)_/¯

[-] Ascrod@midwest.social 9 points 1 year ago

Shit happens, thank you for reverting it and communicating it. :)

[-] Wander@yiffit.net 8 points 1 year ago* (last edited 1 year ago)

Anyway us other instances can help? We still have copies of the content (except for images).

We should also have the activitypub objects that we've received cached in our database.

Edit: it seems that my instance fetches the latest posts instead of using its cache when loading a midwest community. But still, we should have some content in the activitypub table.

[-] seahorse@midwest.social 5 points 1 year ago

I'm not sure what I could do with it unfortunately. If it can recover itself, that would be awesome.

[-] Wander@yiffit.net 6 points 1 year ago

I don't think there's an easy way to "replay" them. But in theory, you should be able to take the entries in that table related to midwest social from any other instance and start broadcasting them anew. The remote instances will reject them because to them they are duplicates, but you would be able to recover lost content.

Now I realize this is far more complex, but in theory it should be possible to create a tool that does this specifically for these scenarios.

I'm sorry this happened to you :(

[-] seahorse@midwest.social 6 points 1 year ago

Thanks. Yeah, this is what happens when my need to try everything to resolve an issue gets the better of me too fast. My filesystem was 90% full and I didn't want it to run out, so I deleted something I shouldn't have. If you have any idea why my disk would be using up lots of storage even though I'm using an S3 bucket and have my logs limited let me know.

[-] Wander@yiffit.net 5 points 1 year ago

Yes, that might actually be the activitypub table in the database. You can safely delete older entries, like from two weeks ago or older. Otherwise it just keeps growing with the logs of all activitypub objects that the server has sent out received.

[-] seahorse@midwest.social 2 points 1 year ago

Do you know where I can find the query to do that? Databases are not my forte.

[-] Wander@yiffit.net 4 points 1 year ago* (last edited 1 year ago)

Yes, let me logging to my server and try to retrieve the exact query I used. BRB

[-] SmokeInFog@midwest.social 1 points 1 year ago

W3 Schools entry on SQL DELETE statement

I don't know the actual table definitions for lemmy, but it should look something like:

DELETE FROM activitypub
WHERE createdDate < ((SELECT CURRENT_DATE) - interval '2 weeks') 
[-] trafguy@midwest.social 5 points 1 year ago

Thanks for the transparency! I don't mind, mistakes happen, but I understand it's frustrating and a bit problematic with the lost content.

There was a post about that on a Lemmy admin community a few days ago. Someone with a ~1k userbase was eating up a GB/day on average. IIRC, there were lots of logs, but also if I understand correctly, every server stores mirrors of the data from anything users subscribe to. That could eat up a lot of data pretty quickly as the fedeverse scales up.

If you wanted to suggest a shift for improved scalability, maybe servers could form tight pacts with a few who mirror each other's content, and then more loosely federated servers load data directly from the pact. A compromise between ultimate content preservation and every larger server having to host the entire fedeverse.

So basically, a few servers would form a union. Each union would be a smaller fedeverse (like cells in an organ), and they'd connect to other organs to form the fedeverse/body.


Also, are users who joined in the past few days affected? I suppose they might need to sign up again.

[-] seahorse@midwest.social 3 points 1 year ago

Yeah, if they signed up in the last few days they'll need to do it again. Ugh.

[-] Nemo@midwest.social 8 points 1 year ago

Was just coming here to ask what happened, thanks for the update.

[-] SmokeInFog@midwest.social 7 points 1 year ago

I might be in danger of being "that guy", but how come you're not doing more frequent backups?

[-] seahorse@midwest.social 5 points 1 year ago

Digital ocean does a backup once a week over the weekends. That's just the schedule I'm on. I suppose I could do my own but that would require finding somewhere to store 60 GB. I'll look into it though.

[-] mouse@midwest.social 5 points 1 year ago* (last edited 1 year ago)

I can recommend Hetzner's storage boxes: https://www.hetzner.com/storage/storage-box?country=us 1TB for backups costs less than $4 a month.

It's not the fastest transfer because it's hosted in Germany or Finland.

Note: It is in EUR will be converted from USD.

[-] Wander@yiffit.net 5 points 1 year ago

What I do is create a new live snapshot before doing any changes to the server. I don't use digitalocean, but I think their snapshots cost 6 cents per GB/month, and are only charged like that if you retain them the whole month.

If everything goes well after a change and you delete the snapshot a day or two later (or after the next backup), it's basically free or very cheap. I think it's just a minimum of $0.01 per snapshot.

[-] andrew@midwest.social 6 points 1 year ago

Thanks for the update. I do still see my posts if I visit them in their source instance like here for example https://lemmy.ml/comment/1654092

[-] Redpandalovely@midwest.social 3 points 1 year ago* (last edited 1 year ago)

It kind of sucks, but it's ok.

load more comments
view more: next ›
this post was submitted on 19 Jul 2023
79 points (96.5% liked)

main

1335 readers
5 users here now

Default community for midwest.social. Post questions about the instance or questions you want to ask other users here.

founded 3 years ago
MODERATORS