this post was submitted on 04 Aug 2025
55 points (100.0% liked)

Technology

350 readers
151 users here now

Share interesting Technology news and links.

Rules:

  1. No paywalled sites at all.
  2. News articles has to be recent, not older than 2 weeks (14 days).
  3. No videos.
  4. Post only direct links.

To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:

More sites will be added to the blacklist as needed.

Encouraged:

founded 3 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] CarbonatedPastaSauce@lemmy.world 9 points 1 week ago (1 children)

The only surprising thing to me from this article is that OpenAI actually follows the rules for bot crawlers.

[–] 0_o7@lemmy.dbzer0.com 5 points 1 week ago (1 children)

Or they haven't been caught yet.

The article explains PerplexityBot respects robots.txt, but then sends a different request with a different IP and different user-agent. They could very well be using a different method to walk around it.

The article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.