2535
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 03 Jul 2023
2535 points (98.0% liked)
Technology
59672 readers
3838 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
Doesn’t sound like retaliation to me, it sounds like their scheduled web crawlers are finding that content they used to index is now no longer viewable and this removed from search results. Pretty standard. My guess is that there were 400 million URLs listed and as the crawler uncovers that they are no longer available, that number will keep dropping to reflect only content publicly viewable. If only 500 URLs are now publicly viewable (without logins) then that’s what they will index. Google isn’t a search engine for private companies (unless you pay for the service) they are a public search engine so they make an effort to ensure that only public information is indexed. Some folk game the system (like the old expertsexchange.com) but sooner or later google drops the hammer.
This was my thought too. Their crawler is marking links unreachable as it re-crawls them.