this post was submitted on 17 Aug 2025
705 points (99.7% liked)

Technology

468 readers
279 users here now

Share interesting Technology news and links.

Rules:

  1. No paywalled sites at all.
  2. News articles has to be recent, not older than 2 weeks (14 days).
  3. No videos.
  4. Post only direct links.

To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:

More sites will be added to the blacklist as needed.

Encouraged:

Misc:

Relevant Communities:

founded 3 months ago
MODERATORS
 

Comments

Source.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] Redjard@lemmy.dbzer0.com 9 points 1 week ago (1 children)

Crawl-delay is just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers ...

[โ€“] who@feddit.org 2 points 1 week ago* (last edited 1 week ago)

Crawl-delay

It's a nonstandard extension without consistent semantics or wide support, but I suppose it's good to know about anyway. Thanks for mentioning it.