Building hybrid search over a personal corpus

I keep everything I write. Journals going back to 2012, every conversation I have had with a model since 2024, meeting transcripts, voice notes, book digests. It is a few thousand markdown files and it is useless in the way a filing cabinet is useless, which is that the thing you want is certainly in there and you will not find it by opening drawers.

Motivation

What I want from it is narrow and specific. I want to ask a question in ordinary language, get back the ten passages most likely to answer it, and have that take about a second.

The last part of that sentence is the one doing the work, and it took me a while to believe it. A second feels like an arbitrary target until you notice what happens either side of it. Under a second, searching is something you do mid-thought, the way you glance at a shelf, so you run it on a hunch and you run it five times in a row narrowing as you go. At fifteen seconds it stops being a glance and becomes an errand, which means you only run it when you are already fairly sure it will pay, which means you never run it on a hunch, which is the case where it would have told you something you did not know.

So latency is not a performance concern here, it is the whole feature. A slow search over your own writing gives you the answers you could have guessed. That is the constraint everything below is built against.

Adjacent work: keyword and vector search

Two families of search exist and each fails in a way the other does not.

Keyword search finds documents containing your words. It is exact, it is fast, and it has no idea what anything means, so a search for "forcing function" misses the entry where I wrote "constraint that makes me do the thing". The implementation almost everyone uses is BM25, which ranks a document by how often your words appear in it, discounted by how common those words are everywhere else. Rare words count for more, which is why searching for a distinctive phrase works well and searching for "the system" does not.

Vector search finds documents that mean something similar. Every chunk of text is converted into a list of numbers, a few hundred to a few thousand of them, positioned so that passages about similar things land near each other. A query gets converted the same way and you return whatever is closest. This finds the entry about the constraint that makes me do the thing. It also cheerfully returns things that are merely adjacent, and it cannot find a proper noun it has never seen.

The failures are complementary, so you run both and combine them. Which raises the question of how, because BM25 scores and vector distances are not comparable numbers and there is no honest way to add them.

The trick is to throw the scores away and keep only the ranks. Reciprocal rank fusion gives each document a score based on its position in each list, first place counting for more than second, and sums those across the lists. A document ranked third by keyword and fifth by meaning beats one ranked first by keyword and absent from the other. You never have to decide what a BM25 score of 14.2 means relative to a cosine distance of 0.31, which is good, because nobody knows.

Reranking

Fusion gives a decent ordering cheaply. A reranker gives a better one expensively, so you use it on a small set.

The pipeline retrieves fifty candidates, fuses them, and hands all fifty to a reranking model that reads the query and each passage together and scores the actual relevance. Running that model over the entire corpus would be absurd. Running it over fifty is a network call.

Fifty is a parameter rather than a constant, and so is everything else worth changing:

$ organs cortex search --help

Options:
  -m, --mode <mode>        vector (default), hyde, keyword, hybrid
  -n, --limit <n>          Max results (default: "5")
  -c, --collection <name>  Filter to a single collection
  -p, --path <prefix>      Filter results to path prefix
  -v, --verbose            Show timing breakdown
  --pool <n>               Candidate pool size before reranking (default: "50")

The mode flag is the argument of the previous section made available at the command line, since you can run either half of the hybrid on its own and see what each one finds. The pool size is the knob on the cheap-then-expensive tradeoff: raise it and the reranker sees more candidates and costs more, lower it and you save money by trusting the fusion further. And -v is how the next section exists at all, because it prints the timing of each stage separately rather than the total.

The shape here generalises past search. A cheap method over everything, then an expensive method over what survives, is how you get the expensive method's quality at something near the cheap method's cost.

Measured latency

Measured across five queries, timing each stage separately. These figures come from running the system today rather than from my notes at the time, on a corpus that has roughly doubled since I built it, so treat them as the shape of where the time goes rather than as a benchmark of the March version.

The end-to-end number is the least interesting row. What the stage breakdown shows is where the time actually goes, and four of those stages behave in ways worth knowing.

Reranking is the most predictable thing in the system. It sits at roughly 350ms whether everything else is warm or cold, because it is a network round trip to somebody else's model and it does not care about the state of my laptop.

Fusing is free. The step that makes the whole design work costs less than a millisecond, because it is arithmetic over a hundred integers.

The cold case is ten seconds and it is one line of that table. On the first query after a while, the keyword search takes eight and a half seconds instead of the fifty milliseconds it takes on every subsequent run, and everything else is roughly unchanged. I have not proven the cause, but a hundredfold penalty on the first read of a large index, disappearing immediately afterwards, is what you would expect from the operating system having to fetch it from disk rather than from memory. Whatever the cause, a benchmark that averages cold and warm runs together describes a system nobody uses.

Embedding the query is now the slowest warm stage, at three hundred to eight hundred milliseconds for a network call to an embedding API. When the corpus was small, that call was already the bottleneck: at three or four hundred chunks, scanning every vector by brute force took about one and a half milliseconds while embedding the query took one to two hundred. The scan has grown by three orders of magnitude since and it still is not the slowest part.

Two things called cortex

The name has been on two unrelated pieces of software and I should be clear about that rather than imply a lineage.

The first cortex, in January 2026, was a portable wiki format. A .cortex file was a SQLite database of wiki entries with double-bracket links between them, a command line tool to manage it, a small web server to browse it, and later semantic search over its chunks. I built six of them out of book digests, the largest holding 73 entries and 400 chunks. It was a way to carry a knowledge base around as a single file.

In March I deleted that package and replaced search with qmd, an existing markdown search tool. Two days later I replaced qmd with a new thing I also called cortex, built from scratch, which is the hybrid search described above. The name came back. None of the code did.

I mention the two-day round trip because it is the most useful thing that happened. Adopting qmd was correct and I would do it again: it worked, it indexed everything, and it cost an afternoon. What sent me back was that hybrid queries took fifteen to forty seconds and it carried about two gigabytes of local embedding models to do it. Both of those follow from a decision qmd makes on purpose, which is to embed locally so that nothing leaves your machine. That is the right default for a general-purpose tool and it is a real cost I did not want to pay, since I was already sending text to a hosted reranker and had no privacy left to protect. Forty seconds is not slow, it is a different activity. I stopped running searches, which meant I stopped having the thing the search was for.

Two days of building bought a query I actually run. That was worth it, and I only knew it was worth it because I had run the alternative first.

Implementation decisions

The decisions that consumed the most hours were not the design ones. They were these, and a guide that omits them is describing a build nobody had.

Getting SQLite to load an extension under Bun. The vector search is a SQLite extension, and extensions have to be loaded by the database driver. The usual Node library for this does not run under Bun, and the SQLite that ships with macOS is built without extension loading. The path that works is Homebrew's SQLite plus Bun's own driver.

Choosing the right distance measure, and paying to change it. Text embeddings are normalised to unit length, which makes cosine distance the semantically correct comparison. Getting this wrong does not raise an error. It quietly returns slightly worse results forever, and fixing it meant re-embedding everything.

Making chunking swappable. The corpus keeps acquiring new file types, and the day subtitle files needed to go in, the choice was between special-casing them inside the chunking function or making the chunker a strategy chosen per file type. The second is a smaller change every time after the first.

What it costs to run

The reranker is a paid API call on every search. The embeddings are a paid API call on every search and on every document indexed. Both could be replaced by local models, which is what the tool I abandoned did, and doing so would cost about two gigabytes of disk, a chunk of memory, and the fifteen to forty seconds that made me stop searching.

I would rather pay for the search I run than own the search I do not.