The fog finally broke, and there it was: the end of January.
At this point I was beginning to suspect it had been extended without consultation. Days had started repeating. The same Monday kept turning up in different hats. Even the calendar looked tired of explaining itself.
People spoke of February in hushed, theoretical terms, like a place you might go one day if conditions improved. Some claimed they’d seen sunlight. Others said time was moving forward again, but there was no evidence to support this and these people were rightly ridiculed.
Still, it has arrived. The end of January. The beginning of February. We have made it.
Server Updates
Lemmy.zip
Well January was a fun month. In full "New Year, New Me" mode, the scrapers decided to full on try and take us down in their efforts to steal all the content.
Here is a graph:
Normally, this line should be flat/non existent. The orange is showing how many 502 errors we're seeing, and the purple is when everything gets overwhelmed and we start returning 504 errors and suddenly everyone gets a lovely cloudflare screen as the server screeches to a halt.
Our usual method of dealing with this is working out who is firing all this traffic at us and adding their ASN to the challenge list, which previously has then stopped them in their tracks. However they've likely changed track here, and we were seeing lots of sustained scraping from a whole range of IPs, lots of which were residential, and this left us with really only one choice, which was "challenge everything". So, we did just that.
Can you work out on this graph where we may have implemented that?!
This isn't particularly ideal for any of us but hey, people just can't play nice and we have to do stuff like that to stop us from being overwhelmed. The solve rate (i.e. how many requests are legit) is about 2.75% currently. In our last update we shared a screenshot showing a 0.16% solve rate with about 100k attempts per 24 hours - we're currently seeing 600k attempts every 24 hours, so the magnitude of what we need to deal with keeps going up.
Mods! Want people to tag things in your communities?
The ability to set, edit, remove etc tag enforcement from your communities is now available on modtools.lemmy.zip.
Piefed.zip
We have added the Photon front end to Piefed.zip - photon.piefed.zip It's still in beta due to the ever evolving code over on Piefed, but works really well (just like photon for lemmy does!)
We're also going to schedule some downtime on Piefed.zip to upgrade the server. With all the new users (Hello!!) it's starting to push at the seams of suitable performance, so a little bit of extra juice will be on the way. We'll let you all know when that is due to take place.
Piefed has also picked up a bit of attention on Reddit and the like - so if you've come from there, welcome aboard!
Please Don’t Be a Lurker!
If you’re new here - WELCOME! I hope you’re enjoying your time here :)
Just one small teeny-tiny request. The greatest gift you can give Lemmy.zip and Piefed.zip isn’t money, praise, or a gold plated retirement clock.. maybe. Its participation.
Upvote the things you like. Start a discussion, debate, or ponder about whether cereal is technically a soup. Make new communities if you don’t see one that fits your oddly specific niche obsession. (We don’t judge. Well, we try not to judge anyway.)
The fediverse naturally ebbs and flows, tides of people come and go. But if you’ve found yourself oddly attached to this strange little corner of the internet? Wonderful. Help it breathe. Help it grow. Help it be just a tiny bit weirder in the best possible way.
So if you’ve gone to the effort of clicking Sign Up and proving you’re not a robot (unless you are, in which case hello and welcome to our new AI overlords), then please, I beg of you:
Stick around. Add your voice. It really does make this place better.
Donations
I come with a blunt request this month, one that I'm not a fan of doing and have tried to avoid as much as possible. Please remember that unlike traditional social media sites, we don't sell your data to advertisers to shove adverts down your throat, and we go out of our way to prevent any form of advertisement making its way on here.
This form of social media is truly free to use.
We can only continue to exist because of the generous donations of our users. If you too truly value your freedom, please help support it for everyone with a small donation.
Want to support us? We have a range of donation options to suit you!
Liberpay (Card, Paypal)
Liberpay is our newest donation method. You can donate with Paypal or with a card. The link is https://liberapay.com/dotzip
OpenCollective (Card)
Our classic donation option, but only supports card payments. The link is https://opencollective.com/lemmyzip
If you're enjoying Lemmy.zip or Piefed.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the sites and keeping the virtual lights on.
Ko-Fi (Card, Paypal)
We also have our Ko-Fi page if you'd rather use this site, which also supports Paypal and Card payments.
Thank you!
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Graphs!
Piefed.zip
CPU over 7 days:
RAM
Images:
Cloudflare overview (30 days)
(We did some work on caching and doubled how much we cache!)
Requests
Bandwidth
Unique Visitors
Lemmy.zip
CPU over 30 days:
RAM:
Disk space:
Images:
Cloudflare Overview:
Requests:
Bandwidth:
Visitors:
Stuff for nerds
If, like me, you really like using the terminal on your arch install (I use arch btw) then you will know just how cumbersome it is having to open a web browser just to browse social media and post your thoughts.
Not so long ago, I stumbled across Stegodon which is an SSH-first federated blogging platform - yes, social media in your terminal.
To that end, I've spun up my own personal project at Stegodon.zip which you can connect to via ssh stegodon.zip - it really is that easy (You will need ssh keys first.. you have ssh keys right??)
You are all invited to try it out :)
OK well that's it for now. It has started to snow outside so I'm off to look out the window and moan about the snow while the whole country grinds to a halt, the schools are shut down, flights are cancelled, and supermarket shelves are emptied because there is a very light dusting. Can't wait.
Speak soon!
Demigodrick
Thank you so much for all that you do for this instance, and by extension the Fediverse at large! We all really appreciate you :)
Disk space utilization is represented as a graph of percentage - if you don’t mind sharing, what is the actual storage footprint of this instance at this point in time? If applicable, how much of that is content native to Lemmy.zip versus cached content from other federated instances?
The actual lemmy database sits around 110GB at the moment. And that's with almost 3 years of posts/comments from all over the fediverse, so I think that's a reasonable storage amount.
I don't know of a way of seperating out local vs federated in terms of content to be able to check it's size, it's all held in the same table(s) with a local flag if local. There is probably a way, I'd need someone smarter than me to work it out though.
A surprising risk to storage at one point was prometheus and keeping hold of massive amounts of log files so that we could work out when and where we were being attacked from. By implementing a full challenge in cloudflare, we've been able to stop that which is why that graph comes down a bit.
The biggest storage issue is images, which is 2.1TB at the moment. There is a way to clear this up a bit with some software, but it's apparently got risk, so low down the priority list at the moment as object storage is cheap and doesn't affect the server directly.