lazyadmin

cross-posted from: https://lazysoci.al/post/25567708

I shall shortly be updating Lemmy to 0.19.11 which fixes a security issue. Downtime is expected to be minimal.

At the same time I shall also be updating all alternative frontends to their latest versions.

I shall shortly be updating Lemmy to 0.19.11 which fixes a security issue. Downtime is expected to be minimal.

At the same time I shall also be updating all alternative frontends to their latest versions.

Update

Upgrade completed, we are now on Lemmy 0.9.11! 👍 The update took 6 minutes, which is 5 minutes longer than I had expected. The update included a refresh of database indexes which wasn't documented in the upgrade notes. Sorry for the inconvenience if you tried to access during that period.

cross-posted from: https://lazysoci.al/post/24004627

Along with the upgrade to 0.19.10 today I have made the following additional changes

• Added Photon frontend
  • Based on the project at https://github.com/Xyphyn/photon
  • Available at
    • https://photon.lazysoci.al/
    • https://p.lazysoci.al/
• Added Voyager frontend
  • Based on the project at https://github.com/aeharding/voyager
  • Available at
    • https://voyager.lazysoci.al/
    • https://v.lazysoci.al/
• Added Alexandrite frontend
  • Based on the project at https://github.com/sheodox/alexandrite
  • Available at
    • https://alexandrite.lazysoci.al/
    • https://a.lazysoci.al/
• Added Tesseract frontend
  • Based on the project at https://github.com/asimons04/tesseract
  • Available at
    • https://tesseract.lazysoci.al/
    • https://t.lazysoci.al/
• Removed the Lemmy UI gearbox debug icon
  • Thanks for pointing this out @Blaze@lemmy.dbzer0.com @Blaze
• Created this community
  • Another blaze idea, keep them coming :)

When this post is 1 hour old, I shall be updating lazysoci.al to Lemmy verison 0.9.10

Downtime is expected to be ~10 minutes or less

This evening I have updated Lemmy from 0.19.8 to 0.19.9

There has been an increasing number of new users. Many of these have turned out to be dishonest trolls and subsequently banned. One of the traits of the internet I guess.

For those that remain, warm welcome :)

I shall be moving the pict-rs data from local filesystem to local S3 this evening.

The site shall be offline for an extended period during this migration. There is ~450G in just over 3 million files currently hosted on the pict-rs instance.

Lemmy and all its containers are on an NVMe backed zfs pool. With pict-rs fast approaching 500G I've been exploring alternatives. So I introduced an S3 service using spinning rust.

I expect the migration to take several hours so will start this just before I head off for bed between 2300-0000 GMT.

Update 2025-01-21 03:09 GMT

Downtime 3h 17m -- Migration completed. Images seem to be working, both new and old. Site is catching up on missed posts.

The new server is prepped and ready to take to the datacentre. I've created a vxlan (mikrotik eoip) to the office so have been transferring services to the new server when I had time over the last few days.

I just performed the transfer of Lemmy to the new server which was an unannounced 2h30m downtime (sorry!). It was a good backup restore test though.

To complete the project, I shall be visiting the datacentre on Thursday, 2025-01-02 to swap the old server for the new. lazysoci.al will be offline on 2025-01-02 from 0830 to 1200 GMT

Apologies for the inconvenience.

Update: server swap completed without issue.

When this post is 2 hours old (2100 UTC) I shall be performing maintenance on the pict-rs container and updating lazysoci.al to Lemmy 0.19.8.

During the maintenance period, the site shall be offline.

The Lemmy update should be straight forward, however I also need to purge old cached images in pict-rs which could take a few hours.

Edit: Site back online

Holidays are coming, and with that, spare time!

I shall be updating the instance over the next couple of weeks to 0.19.8. I'll post an update some hours before the upgrade.

Over the holidays I shall be decommissioning the physical server that this Lemmy instance is hosted on. The new server is being built and will be a Dell PowerEdge R640 running all U.2 NVMe drives. Hopefully, this will make the server the fastest it can possibly be.

Our lemmy instance stopped processing new activity sometime on Monday 2024-11-18 morning.

The root cause remains unknown. Services were online. Database was responsive.

Lemmy server logs were showing the incoming ActivityPub requests, no errors, but no response was being returned to the sender. The system was restarted on 2024-11-19 and processing of requests resumed.

Luckily, the protocol allows for some caching of requests across all servers, so after 30 minutes of heavy load, our server had mostly caught up.

I was away on Monday, and I did notice the issue, but I initially thought it was a problem with my mobile app (recently moved to Boost). I normally view Lemmy sorted by "Top - Last Twelve Hours" and on Tuesday this returned zero results, which prompted a closer look.

I have added additional monitoring to the system, checking for the age of the latest post. I shall now receive an alert if a new post has not been received for 15 minutes. This may result in some false positives if Lemmy is quiet so may adjust this in the future.

Example data:

A status page is available at https://status.lazysoci.al/

Our hosting provider had an outage which lasted ~30 minutes.

I raised a support case as soon as I got the alert as I couldn't access the systems via their link, but the 4G backup confirmed servers were online.