this post was submitted on 06 May 2026

194 points (99.5% liked)

Lemmy.ca's Main Community

4170 readers

4 users here now

Welcome to the lemmy.ca/c/main community!

All new users on lemmy.ca are automatically subscribed to this community, so this is the place to read announcements, make suggestions, and chat about the goings-on of lemmy.ca.

For support requests specific to lemmy.ca, you can use !lemmy_ca_support@lemmy.ca.

founded 5 years ago

MODERATORS

194

Lemmy.ca & piefed.ca now behind anubis (lemmy.ca)

submitted 2 months ago* (last edited 2 months ago) by Shadow@lemmy.ca to c/main@lemmy.ca

46 comments fedilink hide all child comments

Thanks to a particularly annoying botnet, everyone's favorite anime cat girl firewall is now helping protect piefed.ca & lemmy.ca from bots and scrapers.

This is requests per second and these are all thousands of scrapers on residential IPs hammering us:

They'd increase their usage until the site started struggling, then move on. I banned their user agents, but have no interest in a cat & mouse game. Anubis should hopefully keep things running much smoother for everyone.

Let me know if you have any trouble!

top 46 comments

sorted by: hot top controversial new old

[–] Shadow@lemmy.ca 49 points 2 months ago* (last edited 2 months ago) (3 children)

red = obvious bots
blue = bots and users hitting the first anubis page (ie, it's 99.9% bots)
green = users.

[–] Gullible@sh.itjust.works 23 points 2 months ago* (last edited 2 months ago) (1 children)

I don’t go to this school, but I’m gonna check whether I’m caught in your new net!

Edit: wew, I’m not a bot!

[–] Shadow@lemmy.ca 13 points 2 months ago (1 children)

I like your nickname vs username.

[–] Gullible@sh.itjust.works 6 points 2 months ago

Thank you! It felt like just the right amount of chicanery

[–] Tiresia@slrpnk.net 11 points 2 months ago (1 children)

It'll be nice to compare the next week's network traffic to the last one's and (presumably) see the spikes disappear.

[–] Shadow@lemmy.ca 9 points 2 months ago

There's definitely a noticeable drop.

I'm surprised our backend traffic so flat, but I'm assuming it's mostly federation

[–] LillyPip@lemmy.ca 3 points 2 months ago

Contrary to what my teachers tried to teach me, I am a user, mean, and req’d.

Take that, Mister Ecker!

[–] Alfredolin@lemmy.wtf 20 points 2 months ago

Bots can get rekt. I am afraid it's still gonna be cat & mouse game, let's see how long anubis works for us (I also use it for my services).

[–] Evil_Shrubbery@thelemmy.club 18 points 2 months ago* (last edited 2 months ago)

Yay!!

[–] egerlach@lemmy.ca 16 points 2 months ago (1 children)

F the bots. Would like to be able to have nice things. Happy that at least this is the 🇨🇦-made solution (at least the primary dev, anyways).

Does Fedecan have the budget to throw a couple of bucks a month to Xe? Completely understand if not, I've done not-for-profit corps before and I know what it's like. But if the budget is there, spending it on a Canadian dev would be a nice choice, IMO.

[–] Shadow@lemmy.ca 9 points 2 months ago

Oh I didn't realize they were Canadian, we'll discuss!

[–] Limonene@lemmy.world 11 points 2 months ago (3 children)

Name and shame. What are the useragent strings? Can the companies be identified?

It won't affect me personally, because I already hate all AI companies. But maybe I could convince some people if I tell them what a specific company is doing.

[–] whats_a_lemmy@midwest.social 9 points 2 months ago (2 children)

OP says residential IPs :/

[–] phoenixz@lemmy.ca 7 points 2 months ago (1 children)

https://stormproxies.com/ et al are the kinds of site that offer this. Backend accessible rotating residential IP addresses, makes finding the source of the scourge almost impossible

[–] Gullible@sh.itjust.works 4 points 2 months ago

If you really want to get the info, bludgeoning them legally and cheaply with repeated small claims court processes seems asymmetrical enough to become a slightly cash positive hobby

[–] Tollana1234567@lemmy.today 2 points 2 months ago

that make sense, since bots both propaganda ones and the "normal ones" use residential ip on reddit for the same evasion method

[–] phoenixz@lemmy.ca 6 points 2 months ago

Meh, useragents are easily spoofed and something tells me that most (all) AI companies don't really care about behing honest there

[–] Shadow@lemmy.ca 6 points 2 months ago

They're all generic user agents that just look like a browser. Nothing fingerprintable

[–] connaisseur@feddit.org 9 points 2 months ago

On feddit.org it was also implemented to get rid of bots and reduce load on the infrastructure. There had been some complaints because of the anubis landing page initially, however I think the general acceptance of this measure after explaining is rather high.

[–] HubertManne@piefed.social 7 points 2 months ago (1 children)

will I get anime cat girl challenges!?

[–] Shadow@lemmy.ca 7 points 2 months ago (1 children)

Not unless you move over to our servers.

[–] HubertManne@piefed.social 5 points 2 months ago

awww. I was hoping for something better than cars or busses.

[–] Blaze@piefed.zip 6 points 2 months ago

Thank you!

[–] Eyekaytee@aussie.zone 6 points 2 months ago (2 children)

are there any alternatives to anubis?

[–] 9point6@lemmy.world 10 points 2 months ago (2 children)

How come you're looking for an alternative? Does it not do the job for you or something?

[–] Eyekaytee@aussie.zone 7 points 2 months ago (2 children)

tbh i would prefer something silent instead of a full screen block page while it figures out whether I'm a bot or not

I don't even like cloudflare click to confirm you're not a bot pages which auto confirm

[–] panda_abyss@lemmy.ca 8 points 2 months ago* (last edited 2 months ago)

To my knowledge, which is often wrong, that's necessary.

It's a proof of work system, so your browser has to receive the challenge work, create background workers to do it, then submit the results and get authenticated.

If the work wasn't challenging (slow), then it wouldn't be any impediment to scrapers and bots.

Whether there are alternatives to proof of work that work well, I do not know. But fingerprinting alone is actually very difficult.

[–] 9point6@lemmy.world 3 points 2 months ago

FWIW I think cloudflare and similar do the full screen thing too, they just render a blank page though so it just feels like more load time.

I don't run Anubis on my stuff currently, but I'd be surprised if it doesn't have a similar feature

[–] flameleaf@ani.social 2 points 2 months ago

Thunderbird fails the check. I can't access communities through my RSS reader.

[–] solxix@pawb.social 7 points 2 months ago

Look into Iocaine and other similar "ai tarpits."

[–] RickyRigatoni@piefed.zip 5 points 2 months ago (2 children)

Again. Not a cat.

[–] SkybreakerEngineer@lemmy.world 4 points 2 months ago (1 children)

Not a space snake either. Very disappointed.

[–] RickyRigatoni@piefed.zip 1 points 2 months ago

She could be, we haven't seen her back yet.

[–] GreenBeard@lemmy.ca 4 points 2 months ago (1 children)

Chibi jackal girl. Still feels slightly blasphemous.

[–] Evil_Shrubbery@thelemmy.club 3 points 2 months ago

[–] Illecors@lemmy.cafe 5 points 2 months ago

It definitely did help for lemmy.cafe!

[–] flameleaf@ani.social 5 points 2 months ago* (last edited 2 months ago) (3 children)

Is this why Thunderbird is suddenly spouting errors whenever it checks a lemmy.ca feed? I'm not a bot.

EDIT: fixed

[–] Shadow@lemmy.ca 6 points 2 months ago

I didn't realize thunderbird could access lemmy. Will look later today.

[–] Shadow@lemmy.ca 6 points 2 months ago (1 children)

Try now?

[–] flameleaf@ani.social 4 points 2 months ago

It looks like its working now. Thanks for for your help!

[–] Grimpen@lemmy.ca 4 points 2 months ago (1 children)

Wait, Thunderbird queue with Activity Pub or Lemmy?

[–] flameleaf@ani.social 5 points 2 months ago (1 children)

Yep. The community pages all provide RSS feeds and Mastodon has them as well.

It's so nice using it as an all-in-one newsfeed aggregated with everything else.

[–] Grimpen@lemmy.ca 3 points 2 months ago

Now that you say that, I did set up another RSS reader to display a Lemmy RSS. Thunderbird sports RSS, therefore…

[–] quaff@lemmy.ca 4 points 2 months ago (1 children)

I'll be curious to know if y'all experience any federation issues. If not, I may introduce this on the mastodon instances I administrate!

[–] Shadow@lemmy.ca 4 points 2 months ago

Nothing so far. Anubis has a built in rule set for activity pub.

[–] MrDrProf@lemmy.ca 3 points 2 months ago

Hell yeah! You guys are amazing!