this post was submitted on 30 Jul 2026

153 points (96.4% liked)

Selfhosted

61129 readers

252 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

153

Hister: a private search engine [AIP] (github.com)

submitted 3 days ago* (last edited 3 days ago) by asciimoo@lemmy.ml to c/selfhosted@lemmy.world

41 comments fedilink hide all child comments

Hi everyone,

I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.

Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.

Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.

The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.

Some other features:

Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
Optional semantic search using an embeddings endpoint you configure
Persistent website crawls
Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
Web, terminal, command line, HTTP API, and MCP interfaces
SQLite and PostgreSQL support, plus optional multiple user hosting

Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.

The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.

Project: https://github.com/asciimoo/hister

Website and documentation: https://hister.org/

Small read-only demo: https://demo.hister.org/

I'd appreciate feedback, questions, and suggestions as well as joining our growing community.

AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy

all 42 comments

sorted by: hot top controversial new old

[–] curbstickle@anarchist.nexus 17 points 3 days ago (1 children)

@asciimoo@lemmy.ml please update the title with the appropriate tag (rule 7) and if AI was involved in development, add the disclosure per rule 8 (sample disclosures can be found here)

Thanks!

[–] asciimoo@lemmy.ml 12 points 3 days ago

Thanks for letting me know. I've updated the post.

[–] conrad82@lemmy.world 3 points 1 day ago (1 children)

Can i use this in my self hosted environment? the docs talk mainly about running it in the terminal

does it have a docker install?

can i connect it to other services like paperless, or would i need to manually import files?

[–] conrad82@lemmy.world 1 points 1 day ago (1 children)

I have tried it a bit today and I like it, but I have a few questions

Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file
After just playing with it for a day, the disk usage is 150 MB. I figure I should use postgres instead of sqlite. Do you have any examples on how to set that up, preferably with docker? I guess I can figure it out, but it might take me some time. I am not familiar with pgvector

[–] asciimoo@lemmy.ml 3 points 1 day ago (1 children)

can i connect it to other services like paperless, or would i need to manually import files?

Hister supports importing data from a few services, but paperless isn't supported yet. More details: https://hister.org/docs/import

Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file

Docker is perfectly fine. Every settings option from the config file can specified using environment variables. The syntax is HISTER__[SECTION]__[OPTION]=[VALUE].

After just playing with it for a day, the disk usage is 150 MB

Probably most of the disk space is occupied by the Hister binary which contains all the N-grams required to identify ~30 languages. The index should be much smaller.

I figure I should use postgres instead of sqlite.

SQLite is more than enough for personal use, but if you prefer to use postgres, just specify the standard DSN formatted connection data to the server.database config option: https://hister.org/docs/configuration#database-backends

how to set that up, preferably with docker

Use the HISTER__SERVER__DATABASE="host=localhost user=hister password=hister dbname=hister port=5432" environment variable.

I am not familiar with pgvector

Hister automatically creates the database model and handles the migrations if required.

[–] conrad82@lemmy.world 2 points 17 hours ago (1 children)

Thank you for your response 🙂 I will stick to sqlite

How do i use environment variables to make a indexer? I couldn't get it to work

i.e. the part with directories https://hister.org/posts/indexing-local-files-with-hister

indexer:
  directories:
    - path: ~/Documents
      filetypes: ['md', 'txt']
    - path: ~/code/projects
      filetypes: ['txt']
      excludes: [, 'secret/*', '*key']
    - path: ~/notes
      patterns: ['*.org', 'doc_*', 'README']

[–] asciimoo@lemmy.ml 1 points 15 hours ago

How do i use environment variables to make a indexer? I couldn’t get it to work

Repeatable config options are not supported from env vars. In this case you have to mount a config file to your container. But, if you want to track local files from docker, you have to mount those files/directories anyway. I suggest to use the standalone binary release if you'd like to track local files, it is way more convenient.

[–] Ladokaka@piefed.social 1 points 1 day ago

Awesomely cool and federated search would be a boon later! Hope you keep it going!

[–] MoogleMaestro@lemmy.zip 18 points 3 days ago (1 children)

Any interested in making this a federated search?

I've been toying with the idea of a web-ring style search engine where it uses the fediverse's post and sub system so that you can subscribe to "indexers" that are made by other users (or even mastodon users themselves) so that you can have highly targeted search engine systems. So, for example, you could make multiple search "scopes" and each of those scopes would follow specific other "sources" and search from them. Likewise, users could provide links and descriptions to add new entries to each scope which would, in essense, work like a fediverse post and then be distributed to all other interested parties.

I haven't done a lick of implementation yet, but thought I'd share the idea here as I consider doing this more and more every day. The only thing I haven't figured out yet is images, because ideally that would be a special type of scope view.

[–] asciimoo@lemmy.ml 27 points 3 days ago (1 children)

Federated search is the direction I'd like to go when the core is mature enough.

I'm still trying to figure out the best approach to make the federation secure (no accidental private/confidential data leak) and easy to use. Related conversations: https://github.com/asciimoo/hister/discussions/432 & https://github.com/asciimoo/hister/issues/387 . I'd appreciate help to figure out an optimal solution.

[–] goddard_guryon@sopuli.xyz 2 points 2 days ago

I'd made a separate comment asking the same question, but glad to know you're already working on this. Great work btw!

[–] ralim@sh.itjust.works 5 points 2 days ago

I've been running this for a while since I first heard of it, it's already saved me a few times trying to recall a page for something.

I treat it like a more advanced history and it's great!

[–] ozoned@piefed.social 8 points 3 days ago

Already installed and loving this idea.

[–] anon_8675309@lemmy.world 3 points 2 days ago (1 children)

The name reminds me of this crazy nostradamus “documentary “ that used to play on HBO when I was a kid.

[–] irmadlad@lemmy.world -1 points 2 days ago

https://www.youtube.com/watch?v=7BrJlP62L9U

[–] cultist@feddit.dk 3 points 2 days ago (1 children)

Asciimoo always with the cool stuff!

Think I'll try getting it setup on Kubernetes and using it for searching documentation when programming.

Would you be interested in getting a Kubernetes example in the docs too then? If you have any requirements for it let me know.

[–] asciimoo@lemmy.ml 5 points 2 days ago

I'd appreciate it, thanks. No special requirements. Providing sensible defaults and explaining usage/potential customization options would be great.

[–] sbeak@sopuli.xyz 7 points 3 days ago* (last edited 3 days ago) (1 children)

This seems super cool, you can curate your own search index :O

Being able to preview your search results offline is quite neat too!

One question though, how will you solve the issue of "echo chambers", because from my understanding, it looks like it will only present results that you search for / have searched for, meaning for some people, you could be stuck with results that only support your view of the world.

[–] hoshikarakitaridia@lemmy.world 12 points 3 days ago (1 children)

I mean it kinda looks like that's by design currently. It's an indexer for your corners of the web, not a search aggregator to find new ones.

[–] asciimoo@lemmy.ml 12 points 2 days ago

Exactly, the echo chamber phenomenon is mostly problematic for "discovery type" searches while Hister is mainly for "recall type" search.

Implementing federated search/index sharing could be a partial solution to this issue in the long run.

[–] Fmstrat@lemmy.world 0 points 1 day ago (1 children)

Is there a future where Hister takes on the functions of SearXNG? I.E. new content is discovered through APIs but local or previous content is prioritized?

[–] asciimoo@lemmy.ml 4 points 1 day ago (1 children)

Currently it is not planned. Hister guarantees that non of your data/query/metadata leaves the service if you use it. As I see, this is a more valuable and unique feature than having an integrated metasearch. There are already great metasearch solutions and Hister provides an easy fallback to search providers, so in my opinion this direction would be more of a sacrifice than an improvement.

[–] Fmstrat@lemmy.world 1 points 1 day ago

Oh I didnt realize there was a fallback option, that's great.

[–] glizzyguzzler@piefed.blahaj.zone 3 points 3 days ago (1 children)

Oh damn this could replace the bookmakers. I have Linkding with the Linkding Injector extension, but this would be next level.

You can view the saved text too, nice. Would it be possible to attach an HTML file from like single file for sites that have heavy image content as part of the “view” button? That’d completely replace Linkding/Karakeep/Linkwarden use cases for me

Understandable if not, that’s ancillary to the text search focus

[–] asciimoo@lemmy.ml 8 points 3 days ago* (last edited 3 days ago) (1 children)

Oh damn this could replace the bookmakers.

The inspiration for Hister was a bookmarking app, but I realized that I always forget to manually trigger the bookmarking and I miss so many great resources.

Would it be possible to attach an HTML file from like single file

Currently you can import SingleFile HTMLs using the hister import file command, but no further integrations are implemented yet. Although, rendering the exact SinglePage file as a preview can be added relatively quickly. It would be also nice to accept files directly from the SingleFile extension.

Thanks for the good suggestion, I've added it to my TODO! =]

[–] hirihit640@sh.itjust.works 2 points 2 days ago (1 children)

I have tons of webpages saved using the Firefox built-in page saver, which saves and html file and a corresponding folder for the other resources (images, javascript). Would be cool if these could be imported as well. Though maybe the resource folder can be ignored and the html file can already be imported?

[–] asciimoo@lemmy.ml 2 points 2 days ago (1 children)

Though maybe the resource folder can be ignored and the html file can already be imported?

It depends. Hister always requires a unique URL for each document. SingleFile snapshots include the original URL of the document as a meta HTML element. I'm not sure if the built-in page saver provides URL information.

[–] hirihit640@sh.itjust.works 1 points 19 hours ago

No unfortunately it does not. Sometimes the webpage itself contains a meta tag with the original URL but not always

[+] SuspiciousCarrot78@aussie.zone -10 points 2 days ago* (last edited 2 days ago) (1 children)

If anyone else is confused wtf this does, I pointed the clankers at it and asked. Way to bury the lede OP.

See [6] for some missing features (?) that would be nice to clear up.

[1]

You open a webpage normally.
The browser extension reads the page that your browser loaded.
It sends the page text, title, URL, and other details to your Hister server.
Hister stores and indexes that content.
Later, you search Hister for words from the page.
Hister finds the saved page, even when you cannot remember its title or URL.

[2]

so, like a Google search of visited sites and my own drive?

Yes. That is actually a reasonable use case.

If you pointed Hister at:

C:\Documents

it would:

Walk the folder tree.
Read supported files.
Build a full-text index.
Watch for changes.
Re-index new or edited files automatically.

You could then search for things like:

tax invoice lenovo

lumbar radiculopathy

that python script with sqlite

"Methylphenidate" (exact phrase)

without caring which folder the file lives in.

Is it like Google?

Yes, but only for your own data.

Imagine Google, except the only websites are:

C:\Documents

D:\Notes

your browser history

pages you crawled yourself

You open a browser, type a search, and receive ranked results from your own files instead of the internet.

[3]

Can you access it from elsewhere on your LAN?

Yes.

By default it listens only on localhost:

127.0.0.1:4433

However, you can configure it to listen on all network interfaces:

0.0.0.0:4433

Then any device on your LAN can use the web interface or API, provided your firewall allows it. You should also enable an access token or user accounts if you expose it beyond your own PC.

[4] What interests me more for your home setup is that Hister is not just a search engine.

It already exposes:

a web UI,

a REST API,

and an MCP endpoint for AI assistants.

That means you could run one Hister instance on your server, index your documents once, and then:

search them yourself from any device on the LAN,

or let an AI search the same index through MCP.

If I wanted a self-hosted "Google for my own stuff" with minimal setup, Hister is much closer to that goal than Meilisearch. Meilisearch is an indexing engine. Hister is already a complete document search application.

[5]

Yes. By default, Hister stores its data in SQLite:

Search index and metadata: db.sqlite3

Stored HTML previews: separate files on disk

Configuration: YAML

It can also use PostgreSQL instead of SQLite if you want.

Regarding browser history, there are two separate things:

Your browser's history database (Chrome, Firefox, etc.).
Hister's own index.

They are independent after import.

For example:

Monday: You visit example.com.

The extension sends the page to Hister.

Hister stores the text, title, URL, metadata, and optionally the HTML preview.

Tuesday: You clear Chrome's history.

The Hister copy remains. Clearing your browser history does not delete Hister's copy. Hister is designed to be a long-term knowledge store, not a mirror of your browser history.

If you want Hister to forget something, you must delete it from Hister itself. The web UI and terminal both support deleting indexed entries.

I actually like that design. If it deleted documents every time you cleared browser history, it would defeat one of its main purposes. It treats browser history as an ingestion source, not as the source of truth.

[6]

From the current documentation:

Are sources timestamped?

Yes.

Each indexed document records at least:

Added timestamp.

Updated timestamp.

For browser pages, the visit timestamp.

URL, title, and metadata.

Is there a TTL (auto-expiry)?

No.

I could not find any TTL, age-based expiry, or retention policy.

Documents remain until you delete them or overwrite them with newer content.

Can I set a maximum database size?

Not that I can find.

There is no option such as:

10 GB maximum.

Delete oldest first.

Keep only the last 365 days.

Limit to 1 million documents.

You can reduce storage by:

Disabling HTML previews.

Limiting indexed file size.

Restricting indexed file types.

Excluding paths or domains.

Can I cleanly delete by topic?

Partly.

You can:

Delete individual documents.

Delete crawl jobs (but not the indexed documents).

Label imported content.

Query by labels and metadata.

Configure rules to stop future indexing.

I did not find built-in support for bulk deletion such as:

"Delete everything about Tesla."

"Delete all Reddit pages."

"Delete every page tagged AI."

The API may allow scripted deletion, but I did not see a first-class "delete by query" feature in the documentation.

Is browser ingestion automatic?

Yes.

That is one of its main features.

Once the browser extension is installed and pointed at your server:

Every page you visit is captured automatically.

The extension extracts the rendered page.

It sends it to Hister.

If the page later changes, the extension periodically checks it and updates the stored copy.

Duplicate visits update the existing document rather than creating a new one.

You do not need to press a button.

My only concern

This project feels like it is missing a lifecycle policy.

For a personal knowledge base, I would want options such as:

Keep browser pages for 90 days.

Keep GitHub forever.

Keep PDFs forever.

Delete pages that nobody has opened for two years.

Maximum database size: 20 GB.

Delete by label or query.

I could not find those features in the current documentation.

The project is very good at collecting information. It appears less mature at managing its lifetime. That would probably become the first thing I wanted if I intended to leave it running for years.

[7]

Yes, on both counts.

Can I exclude particular sites?

Yes, and it is quite flexible.

Hister has skip rules that apply before a page enters the index. You can exclude:

Entire domains.

Subdomains.

URL patterns.

Admin pages.

Cookie consent pages.

Anything that matches a regular expression.

For example:

^https://reddit//.com/ ^https://mail//.google/.com/ ^https://.*.bank.com/

Those pages never enter the index.

The nice part is that the rules apply both to new browser captures and to re-index operations, so they stay excluded.

How does multi-tenancy work?

It is better than I expected.

Each user has:

Their own login.

Their own search index.

Their own browser history.

Their own skip rules.

Their own search aliases.

Their own API token.

If you and your wife both use the same Hister server:

You ├── browser history ├── indexed pages ├── rules └── API token

Wife ├── browser history ├── indexed pages ├── rules └── API token

Neither user can search the other's documents.

There is also a global area.

You can configure directories such as:

/shared/docs

Those appear in everyone's searches, while personal folders remain private.

One feature I really like

Directory indexing also supports user ownership.

For example:

alice -> /nextcloud/alice bob -> /nextcloud/bob global -> /shared/docs

The server enforces that visibility automatically. Even administrators do not automatically see other users' indexed files.

Overall, I came into this expecting "browser history search". After reading the documentation, it is closer to a lightweight, self-hosted personal search appliance with sensible multi-user support. The remaining gaps are lifecycle management features such as retention policies, TTL, storage quotas, and richer bulk deletion. Those do not appear to be built in.

[–] asciimoo@lemmy.ml 7 points 2 days ago (1 children)

The summary has numerous inaccuracies. Most importantly: it is pretty easy to delete content by topic or age. The hister delete command can accept a search query to remove only matched documents. The same is true on the web UI "actions -> remove all matching documents". You can quickly filter by age, simply query updated:>365d. Combine it with URLs, labels, domains or phrases.

[–] SuspiciousCarrot78@aussie.zone -1 points 2 days ago* (last edited 2 days ago) (2 children)

Excellent - thanks for clearing that up.

Is there a TTL / max database size per user setting? Say I have 4 users using the server; can I allocate a hard limit of 10GB per user, with 180 day retention rules?

Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

I would like to further recommend Hister but your documentation is somewhat confusing at first blush.

[–] ReluctantMuskrat@lemmy.world 0 points 1 day ago* (last edited 1 day ago) (2 children)

Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

This is one of the problems with relying on AI... it can produce an overwhelming amount of content with errors and inaccuracies throughout. If you don't review and know the content yourself, you won't know what it got wrong.

It's genuinely rude to lazily use AI to produce such a large babble of details and then ask someone else to review it for errors when you haven't reviewed it yourself. I know you didn't mean it that way but that's nonetheless the result. People are going to read your comment and be misled about this project all because they assume AI is accurate and you didn't review its results.

Edit: Sheesh... I hadn't even got to your shitty comments that followed. You use AI to make a low effort but highly verbose post and then get mad that the repo author won't review it in detail for you when you can't be arsed with reading the docs yourself.

[–] lambalicious@lemmy.sdf.org 1 points 1 day ago (1 children)

This is one of the problems with relying on AI… it can produce an overwhelming amount of content with errors and inaccuracies throughout. If you don’t review and know the content yourself, you won’t know what it got wrong.

Yep, and this induces anyone who wants to review the resulting slop to need to turn to AI as well to even deal with the amount (or dismiss as a whole).

AI is a viral infection on software development.

[–] ReluctantMuskrat@lemmy.world 2 points 1 day ago

Yep... couldn't agree more. I has its uses but too many people are using it and disconnecting their brain. Going to meetings where people have used it to determine requirements, do analysis or even transform data and then don't even review the results themselves before presenting and asking us to review them is rage-inducing. "What do you mean it has problems? Where? What's wrong??" And there's like 5 things I've spotted in 2 minutes and it's clear they've not even reviewed it themselves.

This guy and his long-ass, poorly formatted AI-generated "documentation" no one asked for and then after numerous errors are pointed out.... "Will you review the rest?" The audacity!! 😄

[–] SuspiciousCarrot78@aussie.zone 0 points 1 day ago* (last edited 1 day ago)

This is my final reply in this thread. The developer has said their piece, and I have said mine. Now you've waded in - so let me set the record straight.

I am a developer. I had genuine interest in this project. I read the Hister documentation and inspected parts of the repository because the documentation did not clearly answer several basic questions I had:

How SQLite, Bleve, and stored HTML relate.
Whether TTL or storage quotas exist.
How browser-history deletion affects stored data.
How previews differ from a real web archive.
What multi-user isolation actually covers.

Yes, I used AI to assemble a plain-language summary and labelled it accordingly. Not everyone keeps the Hister codebase in their head, not everyone talks in code review and if I had these questions, I'm willing to bet others did too. The AI wrote for a lay audience because I didn't ask it to do QA, I asked it to ELI-5.

The summary contained errors. Fine. That's AI for you. However, if neither I nor the AI could find clear answers after cloning the repo, that supports my point about opacity.

At no point did I request a line-by-line audit. “Points 2 and 5 are wrong” would have answered the question.

Declining would also have been reasonable. Hell, side stepping it would have been fine too. Instead the dev decided to note the inaccuracies and rudely brush them off.

Both you and the dev seem to be under the impression !selfhosted is a one way distribution channel.

The developer came here, invited questions, then turned the raw prawn when questions arrived.

I didn't go to their their Github. I didn't abuse them. I genuinely wanted to know more about their project and share it, perhaps even work to help improve it.

They - and now you, ostensibly a happy clapper for Hister - came here.

Your claims about my effort and intent are assumptions followed by personal abuse.

Try and walk a mile in someone else's shoes before calling them low effort and shitty next time.

[–] asciimoo@lemmy.ml 2 points 2 days ago* (last edited 2 days ago) (1 children)

Is there a TTL / max database size per user setting?

As I wrote, you can simply automate deletion by document age. Schedule a delete event on each day with the desired retention time defined as a filter expression. Database size limit isn't available yet.

Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

No, sorry, I don't have time to correct a copy of a multiple screens long AI prompt.

I would like to further recommend Hister but your documentation is somewhat confusing at first blush.

Which parts are confusing?

[+] SuspiciousCarrot78@aussie.zone -7 points 2 days ago* (last edited 2 days ago) (1 children)

I am happy to narrow it further.

I took the time to read the documentation, ask ChatGPT to summarise what I found, and then reduced my follow-up to a simple request:

«Which of points 1–7 are materially wrong?»

That is not the same as asking you to audit "multiple screens" of AI output.

If the answer is "2 and 5 are incorrect", or even "I do not have time to review it", that is perfectly fine.

However, dismissing it as "a multiple screens long AI prompt" does not only not answer the question, it comes off as abrasive.

As for the documentation, the confusing parts are exactly those I listed: retention, lifecycle management, browser ingestion, storage limits, deletion, multi-user behaviour, and, most importantly, what Hister actually is and who it is for.

What's disappointing is not that you disagreed with the AI summary. AIs are idiots.

It that after inviting questions and feedback, your response to a genuine attempt to understand the project is curt dismissal.

The inner workings of Hister may be obvious to you; they are not obvious to others.

The point is that you came here specifically to invite questions and feedback.

"TL;DR" does not encourage the sort of community engagement you ostensibly came here to seek.

[–] asciimoo@lemmy.ml 3 points 2 days ago (1 children)

However, dismissing it as “a multiple screens long AI prompt” does not only not answer the question, it comes off as abrasive.

I think it is more abrasive to copy/paste a poorly formatted LLM output instead of taking the time and summarizing it to a few sentences just as you did in your previous post.

[–] alcea@feddit.org -4 points 2 days ago

If just it was called hipster ~~and utilized a rapper with backwards facing basecap and bling bling as logo~~

Lets see how it stacks up against the big ones