Blahaj Lemmy Meta

Hey all! You asked for it, so here it is. I'ma drop a diagram here and will refer to it throughout the rest of this post:

Our infrastructure

In simple terms, the way it all works is we have our protected servers in the backend.

PERSISTENCE

We run our postgres database defensively with 3 dedicated (metal) servers, the main servers each have 2× 1.92TB enterprise-grade NVMe drives mirrored (RAID-1).

We use patroni + percona postgres to manage the cluster. Each instance has a software watchdog to trip-wire a failure.

Scripts automate promotion to make sure under normal circumstances (healthy cluster) that the main server, our big EPYC 7401P 128GiB DDR4(ECC) i350 NIC is the primary.

We have a second server which is a less-powerful i9-12900K 128Gib DDR4(non-ECC) i219v NIC that acts as a read-only secondary, and gets promoted to primary on failover.

Then there's the little server that replicates for backups and potential dual-failure disaster scenarios, a meager Ryzen 7 3700X 64GiB DDR4(ECC) i210 NIC and it has 2× 8TB enterprise-grade spinny disks striped (RAID-0), to keep the DB and backups safe.

It will become primary if the other 2 servers cannot.

The database servers are not connected to directly by any of our services, there's instead a load balancer in front which targets the 3× server's HAProxy instances that have one port directed at the current primary and another port for the current secondary.

This database infrastructure is the biggest, costliest and most engineered part of our setup (which is very frugal for what it is). It's taken me through a near catastrauphic situation (double server failure). You can ask Ada... I was freaking the fuck out.

Additionally we have a single Redis instance which runs on a quad CPU ARM (Ampere) 8GiB RAM 80GB SSD.

• Primary server: €83.70/m
• Secondary server: €61.70/m
• Backup server: €40.70/m
• Load balancer: €5.39/m
• Redis: €6.49/m
• Total: €197.98/m (AUD ~$350) for the persistence layer.

APPLICATION

We then have a prometheus/grafana monitoring box at another €6.49/m.

Then we have the services which run just like everyone else on various pieces of hardware, in docker containers for most, but some run standalone on their own hosts. Sharkey €24.49, Lemmy €12.49/m, Synapse €12.49/m, everything else smaller (pyfedi, pixelfed, friendica, photon frontend, various different static frontends, Ada's latest project of the week) runs on a single dedicated docker host costing €38.70/m.

That brings us to another €101.15/m for the application layer.

EDGE

Then under that we now have the edge nodes, which are completely standalone and do not have internal VPCs etc, it's TLS in, TLS out for these machines.

Each node is generally around 2× vCPU 1GB RAM 25GB SSD 2TB bandwidth for around €6/m. We have 4, so that's around €24/m all up.

End user flow

So when you vist the site, you will first typically hit our DNS server. Our DNS server uses the requesting IP, or EDNS client subnet provided to work out which healthy node is closest to your region, and gives you back that IP address.

Side note: It's important to realise that up until this point, what domain you requested is known to yourself and the DNS provider you used. If you care about privacy, you should install a pihole (you can install it in a docker container, you don't need a raspberry pi), and set it's upstream DNS to Quad9 with ECS and DNSSEC enabled (or if you're really adventurous, install your own unbound server and bypass intermediaries completely).

We run a dual-stack IPv4/IPv6 network all the way through so if you're on IPv6, you'll get our v6 IPs, and hopefully enjoy a less NATted, wider MTU'd, packet un-fragmented journey through the internet.

Once you have the IP address from the DNS server, you will connect to the caching caddy server on the closest healthy node to your geographical location and it will terminate your TLS session, decrypting your request.

At this point we'll see if we have the asset to the request you made cached locally, if so we can send it straight back to you, super quickly!

If not then it will connect to the upstream server in our core location over a (presumably) much larger, less latent and more resilient trunk pipe than most consumer-grade bandwidth will provide. (Unless you run your own redundant-path dark fibre into your house, which I'm not discounting entirely... I know people.)

The returned response will then be evaluated for cacheability and sent back to you.

The build

We wanted to make sure that if one of these endpoints degrade / get attacked / shut down / die, that we can spin up a new one really quick. So the deployment and configuration is completely managed by an Ansible configuration. The deployment of a single node or even a complete replacement of all nodes takes about 10 minutes.

Our nodes run 2 pieces of third party software, and a few scripts to manage things.

DNS

For the DNS resolution we run gdnsd with 2 plugins, http_status and geoip.

The http_status plugin monitors the health of the other nodes to make sure it's not sending people to nodes that don't respond.

The geoip plugin uses the requesting IP to determine what region/country you're from, and select a priority list of nodes closes to that region. The first healthy node in that list is the node that's selected.

CACHING REVERSE-PROXY

For the web-serving component, we run a custom xcaddy compiled caddy server with a few modules included: cache-handler, otter storage, coraza WAF, ratelimit, layer4, maxmind and crowdsec.

At the moment only the first 3 are in use, but the other 4 are included in case we need to mitigate attacks or other edge cases in the future.

And that's pretty much it!

If anyone wants any help with setting up their own version of this, or needs more details, let me know. I'd be happy to help.

If a lot of people are interested (which I doubt at this stage, but who knows?) I'd even be willing to create a project or make it dockerisable etc, but I suspect that it's something that most people would just use Cloudflare et. al. for, if the privacy aspect wasn't such a concern.

I've recently found myself without much to do (short version: the company that my company was contracting to went into voluntary administration just before Christmas, while Ada and I were away in Melbourne), so I've had a little bit of time on my hands to do some work on the site infrastructure, free from meetings and corporate wankery. YAY!

One of the things I've wanted to do for a while now is setup some form of edge-node caching and geo-DNS to get the various sites we host closer to you folks who use our instances.

And yes, there's Cloudflare... and Akamai... and Bunny.net... however as a safe-haven for vulnerable minorities, and with the geo-political situation the way it is these days, we really need to be super careful about where we terminate your connections. Who are the intermediate people who can see and collect your data. Who can switch our servers off at a moment's notice, suspend the domain names, shut us down?

Until recently we've known that we are slow on the edge, but we controlled all our own hardware, and we've not had the capacity to do much about it.

So over the last few days, I've taken the time to setup a bunch of edge nodes, migrate DNS away from third party providers, move domain name registrars.

The end result is that (with a few minor site interruptions) now we have our own CDN that we control all the way from DNS resolution until you hit the database on our dedicated servers. Your traffic is encrypted all the way through, our core infrastructure isn't exposed to people who sniff around to see who they can try to report us and shut us down, nobody else can see your browsing it in transit, and for people not in or around Finland, it's noticeably faster to load the site and click around.

To make sure you're all fully informed, I'll carefully disclose our decisions and new structure.

Firstly our edge servers are on Vultr and DigitalOcean. These 2 providers from our research seem to be quite neutral and non-politically aligned, and neither one by themselves can take us entirely down, and neither one of them are where our core infrastructure is located.

Secondly our edge locations have been carefully chosen to be regions that are outside jurisdictions where we can currently see political turmoil, overly zealous conservatism and fascist activity. We've chosen Toronto Canada, Sydney Australia, Amsterdam Netherlands and Frankfurt Germany as our edge node and DNS server locations.

Thirdly we've moved our domains into EuroDNS registrar to minimize the chance that the USA pressures companies to take action against our domains. EuroDNS is a large company headquartered in Luxembourg, and with no ties to the US itself, it's parent company or any sibling companies, this gives us comfort that they can resist any political pressure which may be applied.

If there's any interest in how we setup the infrastructure, let me know and I can make a separate technical post about it.

EDIT - here it is: https://lemmyverse.link/lemmy.blahaj.zone/post/36690717

It's called !teenrelationship@lemmy.blahaj.zone and it seems interesting but I wanna revamp it since it is unmoderated and the user is banned.

I meant to make a piefed account a while back and never got around to it. Tried to do so today and when I clicked the link to get my code I got an error instead.

Authentication Failed

No authentication token found. Please make sure you're accessing this page through the proper authentication flow.

I even tried reloading the page after disabling all my plugins and still got the error (although the styling looked a lot better after disabling NoScript, even though I have blahaj marked as allowed anyways.)

If you don't use our Piefed instance, feel free to skip this message, as it doesn't impact lemmy accounts.

Kaity has just upgraded our piefed instance to version 1.4 and imported all of our blahaj.zone emojis so you can now use them as reacts on piefed too!

There’s also a few other nifty features, such as the ability to mark a comment as an Answer, Stack Overflow style, the ability to hide a post from yourself without blocking the account that posted it and the ability to mark and filter AI content.

Feel free to check them out if you have an account on the piefed instance.

You can see the release notes for Piefed 1.4 here https://codeberg.org/rimu/pyfedi/releases/tag/v1.4.0

I wasn't sure where to put this, but it's meta to Lemmy/Piefed so I figured this is as good a place as any.

I've noticed a trend of certain posters submitting several threads or even copies of the same thread back to back. Sometimes they're all about the same topic, sometimes they're different, but it seems like some people do all their posting at once. The result is usually something like 7 or 8 threads in my feed from the same person back to back, often with all of them being just images. So like, 8 images of cakes in a row or whatever.

Does anyone else find this obnoxious? I personally tend to block people who post this way so they don't show up in my feed, but I'm wondering if it's just me. It seems like poor etiquette to me, but I'm curious if I'm alone in this.

See title. I wanna put up a lgbt countryballs underlemmy (or underpie; either of which I'd call lgballt), but on the Voyager app I don't see an option to. Is it perhaps the case that I'd need to ask the admins of this instance? Or is it because of my app?

For those quick off the mark, you may already have seen her name appear in the list of lemmy admins! We've brought Jorunn on board, as she's already staff on our piefed instance, and she has been helping moderate lemmy via our admin bot, which can approve signups, delete spam etc. So, she now has her own keys to the castle to make her life easier!

You can reach out to her for issues around community/instance moderation, regarding our lbz users etc. Kaity and I are still the best contacts for sys-admin related issues however.

Anyway, welcome aboard Jorunn and thank you for taking this on. I certainly appreciate it :)

Example: https://piefed.blahaj.zone/c/science_memes/p/438077/

It still shows up in the community and on the user’s profile, it just says ‘not found’ when clicked on.

Edit: just discovered that for some reason I can access the post on Piefed through the Voyager app, but still can’t through the browser interface.

I'm a trans woman, but not a binary one. I used to be on Reddit, and I hated r/twoxchromosomes. You know they had trans people complaining about the name within a month of being created? They always use the excuse that they're too big to change, but they knew about the problem before they got big.

So when I saw c/womensstuff on Lemmy, I was cautiously optimistic. Women need a space on Lemmy to be women without harassment from cis men. There's a lot of sexism on this site. More than the transphobia, actually, which is a weird dynamic. For the record I'd rather be with the bear than the man, because the bear is probably going to mind its own business.

But I also know how horrible cis women can be to trans people when they're fixated on maleness as the problem instead of patriarchy. J K Rowling was abused by a man, and she chose to deal with her pain by becoming a Death Eater. Excluding men from the women's space is a good idea, but I wanted to see if the mods were woke enough to pull it off without falling into the TERF trap.

Yesterday there was a post to the community that hit the front page, and a lot of cis men not reading the rules. I'm glad those cis men are being told to go away, they were turning the women's community into a men-talking-over-women community. Spaces with more men than women need to have the patience to hold space for women, or they'll end up as spaces with only men.

And I fucked up yesterday. I pointed out that a man might be bigender, and was rightly told by atomicorange, who is smarter than me, that any bigender people on the community should introduce themselves as a woman before introducing themselves as a man.

But then there's thermal_shock...

If you assume anyone who isn't a woman is a cis man, then 99% of the time you'll be right. But then there's the 1% of people who are nonbinary. And spaces with more binary people than enbies need to have the patience to hold space for enbies, or they'll end up as spaces with only binary people.

I'm glad thermal_shock didn't get banned, but I'm a bit upset that they were told to go away. If c/womensstuff is only for women (binary or nonbinary), that makes sense to me. If it's for women and enbies and trans men, that makes sense too.

But if the mods mix up those two things, then enbies who aren't women are in a weird place. Allowed by the rules, but still told by mods to go away.

Society tells us all there are only two genders. A lot of people know that's not true, but they kinda forget it, because they spend all their time dealing with men and women. And that makes enbies wanna hide ourselves away to avoid having to inconvenience the binaries and maybe get in a confrontation. But we need enbies to be visible so that everyone can have experience around enbies and stop forgetting we exist.

I don't think c/womensstuff is a TERF space. I think they just fucked up yesterday, like I did. But I'm worried about the future, because I don't think their mods have been careful enough in how they think. They're mixing up being a women's space and being a general gender minority space. They haven't picked one or the other. And it's making them mess up.

cross-posted from: https://piefed.social/c/fedimemes/p/1470594/you-re-giving-yourself-extra-moderation-load-when-you-make-it-hard-to-read-the-rules

So earlier we were assigned mod for the community "queer defense front" on this instance.

The person didn't ask us to be mod or communicate with us at all, then promptly banned us for asking people kindly to not use ableist language.

The rules clearly said zero tolerance for hate, and ableism is hate to us. Plus it seems the person didn't ban those who were being hateful to us (at least not that we can see in the modlog), so double standards we guess?

If you want something specific you have to actually, you know, communicate.

We don't care about the ban (it's not a community we particularly care about), or whatever, just the sheer strangeness of it all and the fact that the person didn't apply the same rules to others as they did us.

We've blocked them and the community. But yeah, maybe don't assign people mods if you've never even asked them or communicated what you want? That's not a gift, that's a terrible responsibility you haven't given them any information or preparation for.

Sorry if this is not the right place to pitch this. Feel free to take it down.

This Halloween, Friday Oct 31st, is the release date for the new Riftbound game by Riot Games. My interest in the game is from a place of loving card games. I've been a huge card game fan since I played pokemon as a kid. I played alot of Hearthstone in college and after. Then in the last few years I've played Magic the Gathering Commander. Riftbound mechanically seems like a very solid game and I'm super excited about it.

If people would like a place to discuss the game, strategy, post relevant news articles, deck ideas, etc let me know. I think a dedicated community would a great way to achieve that.

If you want to see an example, take a look at the lorcana community I moderate on lemmy.world.

I could make the community or someone else could. But if I make the community, all I ask is for at least one other person to help moderate it. Single moderator communities can run into issues if that one moderator makes a mistake. So to avoid that single point of failure I would like to work with people to make such a community function smoothly. edit: typo

Also a community doesn't have to be the go to solution. We could post Riftbound memes in 196. Thanks for reading this and I'm excited to hear what people have to say. <3

cross-posted from: https://midwest.social/post/37326864

What a victory; good job, Kansas Supreme Court!

was trying to find out if we are intentionally defederated, I don't see any hilariouschaos communities showing up in the Blahaj search, so I assume they aren't federated - just not sure why and was hoping to learn the lore 😅

i've noticed lately that there seems to be federation issues with lemmy.world. i don't receive updates from communities or users there, and looking at my profile from lemmy.world, it's missing a few comments that were posted in the last days, when usually it's much much quicker to update.

even when looking at .world communities on here when not logged in, they seem to be stuck in the past! for example, from here the last post from !games@lemmy.world is from 5 days ago, but on .world there are many more recent posts.

does anyone else have issues with them? are we blocking them, or are they blocking us?

Unfortunately iOS 26 removed the ability (idk if this is true it just stopped working for me) to start a voice memo without conformation. But other than having to active Do not Disturb and pressing confirm the shortcuts are hands free.

When I try to access lemmy.blahaj.zone I get Safari asking if I want to download it (???) and Firefox just times out. The server status on Lestat shows as fine though, and I don’t seem to be having any trouble loading pics from LBZ.

Is anyone else having this issue?

Hey, sorry if this doesn't go here. But my posts and comments seem to be getting no interaction, which is unusual. I was wondering if piefed.blahaj.zone had shadowbanned me for some reason, though I don't know why that would be.

does the piefed instance get updated automagically?

i was just curious cause i always see changes but the version still says 1.0.1-dev. hopefully you're not having to deploy all the time! i was just curious :3

I came back to lemmy after a short break and lemmy.blahaj.zone (the default client) doesn't load, and on photon and alexandritte upvotes are broken

We keep trying, filling in the form and using the code from the link place and it just keeps telling us that there's an error no matter what we do. Could someone or somemany please look into it? Thanks!

So using lemmy via the main web ui is very slow to load, and then sometimes only loads a few elements recently.

Can this be fixed?

For me at least. lemmy.blahaj.zone doesn't load but phtn.lemmy.blahaj.zone does.