516
The text version of Wikipedia*
The images and other media are a hell of a lot more.
it's 102GB with images, 53GB without
I presume this is images directly hosted on English Wikipedia and not the entirety of Commons where the vast majority of images are kept, right?
Wikimedia Commons is 373TB images. https://commons.m.wikimedia.org/wiki/Special:MediaStatistics
So I have to upgrade my NAS again, ay?
You're not already running petabyte NAS??????
Kinda interesting at a broad level ... that there's still something to the efficiency of language.
Sure storage is cheap now, but so much of the calculation of the utility of data in modern tech is the presumption of an internet connection and retrieval of information over the network.
With the internet going to shit in various ways, local or decentralised computing is making more sense, at least depending on your priorities and perspective. And so all of a sudden, storage tradeoffs become a bit more meaningful. Do I need all of the pictures and media ... or would a simple textual description suffice for most instances with high res media available at a more centralised archive if I'm really interested? A picture is worth 1000 words, but takes a hell of a lot more digital storage space!
load more comments
(1 replies)
The 100Gb version mentioned above does only have thumbnails/lowres pictures, yeah. Better than nothing for some types of articles, but not everything. The true text-only version is actually only ~53Gb though.
Some of the high res photos are ridiculous.
Like a 8000x9000 uncompressed image of someone's hand and weighs about 22mb.
I know that because I use a lot of royalty free images.
Is there an index of the images or something like that?
https://commons.wikimedia.org/
The images are categorised and there's a search function.
load more comments
(1 replies)
Without images Wikipedia is a "mere" 22.14gb.
So something akin to this joke image I saw the other day is actually feasible for Wikipedia?
Chatgpt is also probably around 50-100GB at most
Probably a lot less, keep in mind that whenever it answers a question the whole model is traversed multiple times, going through multiple GBs is not possible in the matter of seconds the model answers.
I'd be surprised if it was significantly less. A comparable 70 billion parameter model from llama requires about 120GB to store. Supposedly the largest current chatgpt goes up to 170 billion parameters, which would take a couple hundred GB to store. There are ways to tradeoff some accuracy in order to save a bunch of space, but you're not going to get it under tens of GB.
These models really are going through that many Gb of parameters once for every word in the output. GPUs and tensor processors are crazy fast. For comparison, think about how much data a GPU generates for 4k60 video display. Its like 1GB per second. And the recommended memory speed required to generate that image is like 400GB per second. Crazy fast.
Plus input data?
No, but it's the model after the input that you need.
load more comments
(1 replies)
I mean, you can self-host your own local LLMs using something like Ollama. The performance will be bound by the disk space you have (the complexity of the model you're able to store), and the performance of the CPU or GPU you are using to run it, but it does work just fine. Probably as good results as ChatGPT for most use cases.
We do this at work (lots of sensitive data that we don't want Openai to capitalize on) and it works pretty well. Hosted locally, setup by a data security and privacy sensitive admin, who specifically runs the settings to not save any queries even on the server. Bit slower than chatgpt but not by much
load more comments
(1 replies)
This saved my ass at my engineering chemistry exam (still a requirement, even for software engineers) where only offline tools were allowed. Love Kiwix!
LOL... Malicious compliance at its best....
Aside from the text clarification, this is also only the US version of Wikipedia.
What worries me though is that most videos linked on Wikipedia are hosted on YouTube. That’s a pretty dangerous choke point.
You mean the English version? There is no US version, thank god.
https://simple.wikipedia.org/ *ducks*
Simple wikipedia is the greatest gift the internet has given us and having it associated with the greatest country on earth makes me the proudman.
I never even noticed any videos on Wikipedia. Maybe for some cinema articles.
Videos aren't an essential part of an encyclopedia.
Ten year old me would beg to differ.
Videos turned Encarta 95 from being an encyclopedia to the encyclopedia!
I jest - a multimedia experience helps but I agree that the text knowledge is the big draw.
The real ones remember wandering around that damn maze answering questions while managing limited torches to see the map.
load more comments
(2 replies)
load more comments
(2 replies)
DYK that Kiwix was actually created by Wikipedia? Back in the late 2000s there was this gigantic effort to select and improve a ton of articles to make an offline "Wikipedia 1.0" release. The only remains of that effort are Kiwix, periodic backups, and an incredibly useful article-rating system.
Can you write more about the rating system you mentioned?
- There is a set of criteria to rate an article B, C, Start or Stub. These are called classes. Similarly, articles can be rated to be of 1 of 4 importance values to a particular WikiProject.
- There’s a banner on every article’s talk page. Any editor can change an article’s rating between one of the above classes boldly; if a revert happens, they discuss it according to the criteria.
- Some WikiProjects have their own criteria for rating articles. Some of them even have process to make an article A-class.
- Before this system, Wikipedia already had processes to make an article a Good Article or Featured article.
- With GAs, a nominator should put a candidate onto backlog. Later, a reviewer will scrutinize the article according to criteria. Often, the reviewer asks the nominator to fix quite a bit of issues. If these issues are fixed promptly, or the reviewer thinks that there are only nitpicks, the article passes. If they aren’t fixed in a week or the reviewer thinks that there are major problems, the article fails.
-
- As with other processes, the nominator and reviewer can be anyone, though reviewers are usually experienced.
- With FAs, a nominator brings the candidate to a noticeaboard. Editors there then come to a consensus about whether the article should pass.
- Both processes display a badge directly on passed articles.
- Both processes have an associated re-review process where editors come to a consensus whether the article should fail if it were nominated today
- There’s also an informal process called “peer review”, where someone just puts an article at a noticeable and anyone can comment about its quality.
- Articles are automatically sorted into categories by their rating and importance. Editors usually look at these to decide which articles to focus on nowadays.
and you should donate to wikipedia if you are gonna do that
I couldn't afford to donate for a long time but I used it near daily. So now I do monthly, probably larger than average, contribution to make up for sibs from other cribs that can't afford it. Pay it forward is indeed a golden rule.
Do you wear a cape? Or are you one of those who doesn't wear one?
No cape. I'm brown so I'm on the radar bad enough as it is as soon as I leave major cities lol.
The benefit of text not taking up much space.
Is there a git repo for it or do I have to redownload the whole thing to do an update?
i remember a time when it was only 2gb for all of wikipedia. usain bolt had just burst onto the world stage at the time.
And by now he's exited the solar system at incomprehensible speeds.
I did! I do! Also all public domain books as part of the project Gutenberg
I know there are a few companies working on DNA storage. From the comment below about the entirety of Wikipedia and Wiki Commons, I’d say that’d be a pretty practical thing to store.
Here’s the wiki article about it.
Imagine downloading it just after some troll changed critical information lmao
load more comments
(1 replies)
view more: next ›
this post was submitted on 20 Jul 2024
516 points (99.0% liked)
Today I Learned
17793 readers
7 users here now
What did you learn today? Share it with us!
We learn something new every day. This is a community dedicated to informing each other and helping to spread knowledge.
The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:
Rules (interactive)
Rule 1- All posts must begin with TIL. Linking to a source of info is optional, but highly recommended as it helps to spark discussion.
** Posts must be about an actual fact that you have learned, but it doesn't matter if you learned it today. See Rule 6 for all exceptions.**
Rule 2- Your post subject cannot be illegal or NSFW material.
Your post subject cannot be illegal or NSFW material. You will be warned first, banned second.
Rule 3- Do not seek mental, medical and professional help here.
Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.
Rule 4- No self promotion or upvote-farming of any kind.
That's it.
Rule 5- No baiting or sealioning or promoting an agenda.
Posts and comments which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.
Rule 6- Regarding non-TIL posts.
Provided it is about the community itself, you may post non-TIL posts using the [META] tag on your post title.
Rule 7- You can't harass or disturb other members.
If you vocally harass or discriminate against any individual member, you will be removed.
Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.
For further explanation, clarification and feedback about this rule, you may follow this link.
Rule 8- All comments should try to stay relevant to their parent content.
Rule 9- Reposts from other platforms are not allowed.
Let everyone have their own content.
Rule 10- Majority of bots aren't allowed to participate here.
Unless included in our Whitelist for Bots, your bot will not be allowed to participate in this community. To have your bot whitelisted, please contact the moderators for a short review.
Partnered Communities
You can view our partnered communities list by following this link. To partner with our community and be included, you are free to message the moderators or comment on a pinned post.
Community Moderation
For inquiry on becoming a moderator of this community, you may comment on the pinned post of the time, or simply shoot a message to the current moderators.
founded 1 year ago
MODERATORS