Selfhosted

60366 readers

593 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

What steps can be taken to prevent AI training and scraping of my public facing website? (discuss.online)

submitted 6 months ago by kiol@discuss.online to c/selfhosted@lemmy.world

18 comments fedilink hide all child comments

cross-posted from: https://discuss.online/post/32165111

I realize my options are limited, but what about any robots.txt style steps? Thanks for any suggestions.

you are viewing a single comment's thread
view the rest of the comments

[–] JustTesting@lemmy.hogru.ch 2 points 6 months ago (1 children)

A big issue is that this works for bots that announce themselves as such, but there's lots that pretend to be regular users, with fake user agents and ips selected from a random pool with each ip only sending like 1-3 request/day, but overall many thousands of requests. In my experience a lot of them are from huawei and tencent cloud/ASN

[–] Nephalis@discuss.tchncs.de 1 points 6 months ago

Yes, if that is true (and I am not that suprised about it) it is nearly impossible to block them this way.