882

Google won’t comment on a potentially massive leak of its search algorithm documentation (www.theverge.com)

submitted 5 months ago by Bishma@discuss.tchncs.de to c/technology@lemmy.world

140 comments fedilink hide all child comments

A purported leak of 2,500 pages of internal documentation from Google sheds light on how Search, the most powerful arbiter of the internet, operates.

The leaked documents touch on topics like what kind of data Google collects and uses, which sites Google elevates for sensitive topics like elections, how Google handles small websites, and more. Some information in the documents appears to be in conflict with public statements by Google representatives, according to Fishkin and King.

you are viewing a single comment's thread
view the rest of the comments

[-] jonne@infosec.pub 62 points 5 months ago

You mean hosting your own crawler/indexer? That doesn't really sound like a thing you could do cost-effectively.

[-] interdimensionalmeme@lemmy.ml 62 points 5 months ago

No problem we crowdsource the crawling torrent style.

We outsourced that to google for reasonnable performance reason. But they shit the bed so now there's no choice but to do it ourselves.

[-] bamfic@lemmy.world 11 points 5 months ago

ooh that might be an interesting app to run on veilid

[-] interdimensionalmeme@lemmy.ml 1 points 5 months ago

What is that and how does it apply ?

[-] wanderingmagus@lemm.ee 2 points 5 months ago

Veilid is a peer-to-peer network and application framework released by the Cult of the Dead Cow on August 11, 2023, at DEF CON 31.[1][2][3][4] Described by its authors as "like Tor, but for apps",[5] it is written in Rust, and runs on Linux, macOS, Windows, Android, iOS,[6] and in-browser WASM.[7] VeilidChat is a secure messaging application built on Veilid.[1][4]

Veilid borrows from both the Tor anonymising router and the InterPlanetary File System (IPFS), to offer encrypted and anonymous peer-to-peer connection using a 256-bit public key as the only visible ID. Even details such as IP addresses are hidden.[4]

Source: https://en.wikipedia.org/wiki/Veilid

[-] zutto@lemmy.fedi.zutto.fi 19 points 5 months ago

Surprisingly, it's very doable, requires basic technical knowledge and relatively minimal computing resources (runs in the background on your computer).

https://yacy.net/ Github

I have tampermonkey script that sends yacy to crawl any websites that I visit, and it's keeping up relatively good index for personal use of the visited websites. Combine yacy with ~300gb of Kiwix databases, add searxng as a frontend and you have pretty strong self hosted search engine.

Of course you need to supplement your searches from other search engines, as yacy does not crawl the whole web, just what you tell it to.

I encourage anyone who's even slightly interested on this stuff to try Yacy, it's ancient piece of software, but it still works very well and is not an abandoned project yet!

I personally use Yacy mostly on private mode, but it does have the distributed network there as well. Yacy current freeworld status

[-] jonne@infosec.pub 7 points 5 months ago

Yeah, I guess the P2P component sort of solves part of the issue I was imagining by distributing indexes and crawling. I was thinking that people were trying to run all of Google on a raspberry pi at home.

[-] Finadil@lemmy.world 5 points 5 months ago

This is interesting, have you had it index reddit? I'm just wondering how much storage space the database takes up.

[-] zutto@lemmy.fedi.zutto.fi 3 points 5 months ago

Hi!

Great question! I don't crawl reddit, but this applies to other large sites as well. reddit themselves they have at this very moment banned the ip range where I host my Yacy at (Hetzner). I just looked up from my index that I do have 257k pages indexed from reddit through teddit I used to run, this is from before reddit api-enshittification, going to delete those right now.

And the way how the crawling is done is you define crawling depth, which limits how much content is crawled from the site.

0 crawling depth = only the page you send Yacy to, nothing more.
1 crawling depth = all the links on the page you send Yacy to
2 crawling depth = all links on the page you send Yacy to, and all links on the pages crawled..
3 ...
n ...

... etc.

I have my tampermonkey scripts set to only crawling depth of 1 at the moment (Just set them to 2 actually, kinda curious how much more I will be crawling), I've manually crawled some local news sites as a curiosity at the beginning. And my database is currently relatively small, only around ~86.38 gigabytes according to Yacy. This stores aproximately 2.6 million documents in Yacy's Solr.

Yacy memory & disk usage. Yacy solr index size

Yacy has tons of options for crawling, so you can customize how much it crawls and even filter out overly large sites with maximum number of documents set when you send Yacy there.

Large picture of Yacy's interface for starting a crawl.

The tampermonkey script I've been talking about in these posts, it's very simple script: https://github.com/JeremyRand/YaCyIndexerGreasemonkey

Hit me up if you guys have more questions! I'm by no means an expert on Yacy, but I will do my best to answer.

[-] brbposting@sh.itjust.works 16 points 5 months ago

Right!

Before his company was able to block more of Microsoft's own tracking scripts, DuckDuckGo CEO and founder Gabriel Weinberg explained in a Reddit reply why firms like his weren't going the full DIY route:

“… [W]e source most of our traditional links and images privately from Bing … Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine needs to bootstrap with one or both of them to provide a mainstream search product. The same is true for maps btw -- only the biggest companies can similarly afford to put satellites up and send ground cars to take streetview pictures of every neighborhood.”

Ars

[-] warmaster@lemmy.world 16 points 5 months ago

Federated bookmarks?

[-] Paradox 54 points 5 months ago

Federated directories. We're going back to Yahoo like it's 1995

[-] NoIWontPickAName@kbin.earth 32 points 5 months ago

Webrings!!!

[-] AbidanYre@lemmy.world 19 points 5 months ago

<under_construction.gif>

[-] MajorHavoc@programming.dev 4 points 5 months ago

Uh...I know we're all just having fun here, but I need to be part of a webring again. If anyone is more than joking, I kinda need to know about it. Thanks.

[-] yukijoou@lemmy.blahaj.zone 5 points 5 months ago

there are tons of webring still going these days!

[-] MajorHavoc@programming.dev 4 points 5 months ago* (last edited 5 months ago)

Seriously? Cool. I'm going to go do some research then. And maybe entirely change the purpose of my blog, just to fit into one...

[-] mil@sh.itjust.works 1 points 5 months ago

can you share a link to it if you're comfortable with that

[-] harrys_balzac@lemmy.dbzer0.com 4 points 5 months ago

I loved Geocities!

[-] neblem@lemmy.world 6 points 5 months ago

Neocities is trying to be a modern reincarnation https://neocities.org/

[-] NoIWontPickAName@kbin.earth 2 points 5 months ago

I mistook that as neopets

[-] oce@jlai.lu 1 points 5 months ago

Yahoo patiently plotting its return from Japan.

[-] Audacious@sh.itjust.works 7 points 5 months ago

I'm so ready for something like this. I've cleaned up my bookmarks and been waiting for alternatives to search engines.

[-] 4am@lemm.ee 4 points 5 months ago

SearxNG

[-] Cheradenine@sh.itjust.works 7 points 5 months ago

You could use Common Crawl, it's run by a non profit

https://en.wikipedia.org/wiki/Common_Crawl

[-] Im_old@lemmy.world 5 points 5 months ago

Look up the yacy repo in github

this post was submitted on 28 May 2024

882 points (99.8% liked)

Technology

59174 readers

1654 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS