557

Update on SJW upgrade to v19.3 (sh.itjust.works)

submitted 8 months ago* (last edited 8 months ago) by TheDude@sh.itjust.works to c/main@sh.itjust.works

123 comments fedilink hide all child comments

Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

Dual Xeon 2.9Ghz CPUs (32 cores total)
256GB ram
4 x 1TB SSD in raid 10 (with room to add 6 more disks)
10gbit networking

While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.

Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.

Here's the tentative timeline:

Task	Date	Expected Downtime
Migration to new server	Tuesday February 27 2024 @ 8:00PM ET	90 Minutes
Upgrade to V19.3	Thursday February 29 2024 @ 8:00PM ET	Up to 120 Minutes

If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

top 50 comments

sorted by: hot top controversial new old

[-] idefix@sh.itjust.works 100 points 8 months ago

So you bought a server for us? I really need to contribute financially

[-] loutr@sh.itjust.works 32 points 8 months ago

Is there a way to donate yet?

[-] Barbarian@sh.itjust.works 38 points 8 months ago* (last edited 8 months ago)

Highly doubt it. What he has said he needs are volunteers to help run the place

[-] can@sh.itjust.works 10 points 8 months ago

Not as of yet. Leaving me no real avenue to support Lemmy. Does anyone know if there any alternative collectives of developers working on the main Lemmy code I could donate to?

[-] sugar_in_your_tea@sh.itjust.works 15 points 8 months ago

Aside from the main devs? Because you can donate to the core devs on the main website.

load more comments (1 replies)

[-] Vertelleus@sh.itjust.works 70 points 8 months ago

Every time I see SJW, I think "Social Justice Warrior" not "Shit Just Works."

Omnissiah's blessing on the upgrade @TheDude!

[-] Classy@sh.itjust.works 28 points 8 months ago

TIL this instance is not "Shhh, it just works!" and it is in fact "Shit just works"

[-] threelonmusketeers@sh.itjust.works 18 points 8 months ago

I love your alternate interpretation. There are few things in my life which work but I'm not sure how, and I don't want to investigate too much for fear that I break them.

[-] can@sh.itjust.works 11 points 8 months ago

You're just too classy

[-] p1mrx@sh.itjust.works 25 points 8 months ago

I would suggest shijw as an abbreviation.

load more comments (4 replies)

[-] UNWILLING_PARTICIPANT@sh.itjust.works 20 points 8 months ago

I like it. I'm just waiting for someone to bring it up as a mark against our instance somehow

[-] Outtatime@sh.itjust.works 18 points 8 months ago

I've already been accused of being a troll alt right Nazi for using this instance. Weird

[-] UNWILLING_PARTICIPANT@sh.itjust.works 16 points 8 months ago

I thought this instance was nice! That is weird

[-] MomoTimeToDie@sh.itjust.works 10 points 8 months ago

Let me guess, a hexbear user said it?

load more comments (3 replies)

[-] Cracks_InTheWalls@sh.itjust.works 15 points 8 months ago* (last edited 8 months ago)

Does anyone else feel strangely old reading this comment, despite the fact that sh.itjust.works has only been around for less than a year?

"I was there, when the wall came down and the hexbears came"

(Obligatory "sh.it.heads for life"; "up with sh.it.heads, down with shitheads"; and so on)

[-] imaqtpie@sh.itjust.works 13 points 8 months ago* (last edited 8 months ago)

Yeah same. Makes me feel warm inside to know that there are some sh.itheads who never got caught up in the various internet slapfights that have unfolded over the past year.

I mean forget about hexbear, OG sh.itheads will remember the exploding-heads defederation drama. I'll never forget when these users absolutely eviscerated me regarding our "glacial" response to defederating EH.

https://sh.itjust.works/comment/512007

Oof

Your "oh maybe, I kinda guess it's possible" attitude here is the problem. You are acting now, great, but this was an incredibly slow response, it doesn't matter if your vote is tomorrow. It's already incredibly, mind bogglingly slow, and you still decided to post a day long debate thread before the vote (it seems to show overwhelming support for defederation, which I consider totally unsurprising, and makes me wonder again what you've been waiting for).

I understand a little about why it was so glacial, but you continue to appear to not understand that this was actually a huge issue, that in that time you let the server bias towards the alt-right, showed a general support for alt right topics, and alienated an unknowable number of users in the process. Those are all the sort of users you probably wanted to keep, the ones who aren't just content to see things go badly and will take action when they see it rather than stare like a deer in headlights. If they're like me, they were probably posting and trying to find out the delay before leaving in disgust. They're folks that care about having an instance that reflects their values, and you've let them - us - down.

That can't be undone, but the way you keep shrugging it off as though you really don't see it does not inspire confidence.

Even rereading it now, that one still hurts. Thanks for not turning into a bunch of Nazis, that would have been a lot of guilt for me to work through 😅

load more comments (9 replies)

load more comments (6 replies)

load more comments (3 replies)

[-] JohnWorks@sh.itjust.works 54 points 8 months ago

Some reason it never crossed my mind that someone could run their own instance on their own hardware and not like a cloud provider. That's super cool!

[-] InEnduringGrowStrong@sh.itjust.works 36 points 8 months ago

Like Batman and Bruce Wayne, TheDude and TheCloud are never seen together.

[-] brbposting@sh.itjust.works 26 points 8 months ago

Big D Energy

(Dude)

(Decentralized)

(inDependent)

[-] brbposting@sh.itjust.works 39 points 8 months ago

[-] sunaurus@lemm.ee 38 points 8 months ago

Good luck with the upgrade!

[-] merc@sh.itjust.works 32 points 8 months ago

Thanks for running this place, and thanks for the info.

[-] DannyBoy@sh.itjust.works 28 points 8 months ago

What do you expect me to do for that 90 and 120 minutes??

[-] flambonkscious@sh.itjust.works 28 points 8 months ago

Prepare the memes for deployment!

load more comments (2 replies)

[-] willya@lemmyf.uk 27 points 8 months ago

Good luck. Yours brought me to Lemmy and then I made my own soon after.

[-] can@sh.itjust.works 15 points 8 months ago

With a url like that how could you not?

[-] Cracks_InTheWalls@sh.itjust.works 15 points 8 months ago* (last edited 8 months ago)

Godspeed, f.ukwad.

(Hmm...not as cute as sh.it.head. I still kinda like it, though. Excellent domain name choice in general.)

[-] can@sh.itjust.works 23 points 8 months ago

Thank you for all your time and effort.

[-] Cracks_InTheWalls@sh.itjust.works 23 points 8 months ago

The Dude, as per usual, abides.

Thank you again for all you do.

[-] jballs@sh.itjust.works 22 points 8 months ago

[-] Lucidlethargy@sh.itjust.works 22 points 8 months ago

Thank you for everything you are doing. Take a few days if you have to fit this migration and update. I've hosted many servers... I always feel I need to do so much for everyone. Curb that thought, if you can. We genuinely appreciate what you're doing here.

[-] doctorcrimson@lemmy.today 21 points 8 months ago

Lol SJW

[-] cantstopthesignal@sh.itjust.works 11 points 8 months ago

SJWs are ruining everything

[-] Stanley_Pain@lemmy.dbzer0.com 18 points 8 months ago

Not on your instance but good luck!

[-] minibyte@sh.itjust.works 18 points 8 months ago* (last edited 8 months ago)

I appreciate everything you do for this instance. Also, thanks for posting the specs of the new build. It’s pretty neat to see what’s going on behind the curtain.

[-] Snoopy@sh.itjust.works 16 points 8 months ago

Good luck ! We did it for jlai.lu, it was a long week-end :)

[-] SorteKanin@feddit.dk 16 points 8 months ago

Good luck - I previously moved Feddit.dk to a new server and recently upgraded to 0.19.3 without any problems during either upgrade - hope you have the same experience!

[-] CaptDust@sh.itjust.works 16 points 8 months ago

Wow, so new and snappy, smells like fresh cut mahogany in here

[-] nkat2112@sh.itjust.works 15 points 8 months ago

We can't thank you enough and we all love you. You're a rock star.

[-] catsarebadpeople@sh.itjust.works 15 points 8 months ago

Thank you. We love you very much!

[-] Ghyste@sh.itjust.works 14 points 8 months ago

Thank you for everything you do!

[-] AlligatorBlizzard@sh.itjust.works 13 points 8 months ago

Dude, you're the Scotty of the fediverse, you rock.

[-] HerbalGamer@sh.itjust.works 13 points 8 months ago

Thanks Dude

[-] rowrowrowyourboat@sh.itjust.works 12 points 8 months ago

Question. Why would you go with 1tb ssds instead of larger hdds? Isn't the space and price more important than the speed for this use?

You could get double the space (2tb hdd) for the same price as a 1 tb ssd.

Just wondering.

[-] TheDude@sh.itjust.works 28 points 8 months ago

The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.

load more comments (1 replies)

[-] 4am@lemm.ee 20 points 8 months ago* (last edited 8 months ago)

Speed is usually the reason. SSDs in general are faster, enterprise SSDs are not only faster but much more write-tolerant and last a very long time in comparison to consumer SSDs.

They can also (in many cases) do write caching at the speed of a DRAM buffer, making the bottleneck the SATA or SAS bus itself (SAS is like enterprise SATA, 12Gb/sec as opposed to 6). NVMe can be even faster. This means that programs (ie Lemmy and its database) that write data aren’t waiting around for the drive to acknowledge the write before that program can move on to other things. Shaving off a few milliseconds per write can make a massive difference when you realize there might be millions of IOPS (Input/Output operations Per Second) under load. The requirement for low latency is everything in servers.

When you are running a public service and requests are coming in constantly and at a high rate, you really really do not want storage latency to bottleneck you, as that is a problem that will compound extremely quickly. This is a big issue with HDDs as well, as even disk seek times add to the problem, let alone caching/buffering writes.

We could talk all day about if four SSDs in a RAID 10 are optimal, but sometimes you have to think about budget and complexity as well. For the load that a popular Lemmy instance might currently draw, I’d make an educated guess that this might be sufficient for now. Room to expand was also mentioned, which is the second most important part of a storage plan.

load more comments (3 replies)

load more comments (2 replies)

[-] threelonmusketeers@sh.itjust.works 11 points 8 months ago

this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected

Woohoo, that's awesome! Thank you so much!

load more comments

this post was submitted on 23 Feb 2024

557 points (97.8% liked)

sh.itjust.works Main Community

7585 readers

3 users here now

Home of the sh.itjust.works instance.

founded 1 year ago

MODERATORS

TheDude@sh.itjust.works

manifex@sh.itjust.works

biela@sh.itjust.works

imaqtpie@sh.itjust.works