Searching my own journals

Motivation

By August 2024 I had four months of journal entries in a database, written by typing and talking into a chat app on my phone, and a command on my laptop that would answer questions about them. The command worked. It was also the most expensive possible design, and I knew that within a day of building it.

What follows is four versions of the same tool. Each one exists because the version before it hit a limit I could name, and the limits turn out to be more interesting than the tool.

The fourth row is where I stopped, and its limit is still there.

The dump

The first version had no retrieval in it at all. I picked a date range, concatenated every entry in it, put the whole thing in front of the model with my question at the end, and read what came back.

This is worth being precise about, because it is what most people build first and it is not stupid. The model sees everything, so nothing relevant can be missed, and there is no machinery to get wrong. What it costs is that you pay for the entire corpus on every question, and the price goes up every day you keep writing.

I wrote down the problem in early August, and there were two halves to it. The queries were expensive in tokens because there was no retrieval, and they were also not very good at producing high quality output from short questions. The second half is the one that made it a design problem rather than a bill. If shoving everything in had produced excellent answers I would have paid and moved on.

Retrieval

The obvious fix is retrieval: instead of sending everything, find the handful of entries that bear on the question and send only those. Which requires a machine-checkable definition of "bear on", and the standard one is geometric. You cut the corpus into pieces and hand each piece to a model that returns a long list of numbers, arranged so that pieces about similar things come back with similar numbers. Do that to the question as well and the pieces you want are the ones whose numbers sit nearest to it, which turns a question about meaning into a question about distance. That substitution is doing all the work, and it is worth holding onto, because the place it breaks is the last section of this post.

I tried to do this with Azure AI Search and gave up after two hours, on developer experience rather than capability. The note I wrote that afternoon says the developer experience was bad, that learning the whole surface was going to be a headache, and that I would rather push forward on development as normal. Two hours is not a fair trial of a product and I am not offering it as one. It is a fair account of how much patience I had for a setup step in the middle of a different project, which is the situation most tools are actually chosen in.

Precomputed summaries

The replacement appears in the same entry, a few lines later, and it is better suited than the thing it replaced. Rather than index the raw entries, I would run an extraction prompt over each day, one pass per day, and keep the output. Fabric's extract-wisdom prompt did the work. That gives you a small, dense summary of every day that you can search cheaply, and it is the first real architectural move in the whole thread: precompute the summary, then search the summary rather than the source.

It arrived, as these things do, from giving up on the expensive option.

Query cost and question shape

The version with summaries was better and I still could not get anything surprising out of it, and in late September I worked out why.

Without precomputation, every question starts from the raw material. I have to make a model call against unrefined entries, which means I recompute from scratch everything I might have found useful, every time. Which makes sense if you have no idea what would be worth keeping. It buys you total flexibility, because nothing has been decided in advance and any question is equally available.

That flexibility severely limits your imagination, because you are always bounded by compute.

Which sounds like a complaint about money and is not. Follow what a person actually does when they sit down in front of a system like this. They do not enumerate every question they might ask and then filter for affordability. They think of a question, and the ones that come to mind are the ones the system has taught them it can handle, because a tool that has been slow and shallow ten times running has trained you not to bother asking it anything deep. So the cost of a query does not sit downstream of the question. It sits upstream of it, shaping which questions occur to you at all, and the shape of what you can afford quietly becomes the shape of what you think to want.

Precomputation is not an optimization in that light. Moving work before the question does not just make the same questions cheaper, it changes which questions are thinkable, because the expensive part has already happened by the time you are wondering what to ask.

Adjacent work: where journal tools converge

The same night I wrote down that a search over your own journal is the crab.

Crab-like body plans have evolved independently at least five separate times among the decapod crustaceans, in lineages that are not closely related. Flat, wide, folded tail, walking sideways. Nobody planned it. It keeps happening because for an animal of roughly that size doing roughly that job, the crab shape is a local optimum that a lot of different starting points slide into.

Journal tools do the same thing, and once you have seen it you can predict it. Start anywhere, keep improving, and you converge on a search box with a small research agent behind it that reads a few of your entries and writes you a paragraph. My four versions were four points on that slide. The graph felt like an escape and it is really the same animal with a better index.

That is worth knowing early, because it tells you which improvements are free. Anything that moves you along the slide will happen anyway and does not need you. The only interesting question is what sits on top of the thing everything turns into, and I did not have an answer for that in September.

Choosing the unit to precompute

Precomputation only helps if you precompute the right thing, and I had no way to know what that was. A daily extract is one guess. Entities, claims, questions, open loops are others. Each guess costs a full pass over the corpus to evaluate, so the obvious plan is to think hard, pick the best one, and run it. That plan had produced nothing for a month, and the reason it produced nothing is that the thinking had no input. I was trying to reason my way to a unit without ever having looked closely at the operation the unit was supposed to serve.

So I inverted it. Do not jump into the code, and resist scaling too early, because everyone reaches first for the solution that handles the whole database and you have to understand the nature of the problem before going hard at a solution. Instead, mimic the process manually, decompose it into the simplest steps, and ask what is actually going on in your mind when you do it.

The reason that works is not discipline. Doing the thing by hand is the only cheap way to observe the operation you are trying to automate, because your own head runs it for free and a full pass over the corpus does not. A manual run is an experiment you can afford, and the whole difficulty was that you could not afford the experiment.

So I read a few entries with a model, formed hypotheses about how they linked, and built a small corpus of connected ideas rather than a system for building one. What I wanted out of it was already written down in the same note: I wanted to trace back which ideas led to what.

Graph construction

That goal is the reason the answer had to be a graph rather than a better search. Similarity retrieval answers "what else here is about this," which is a question about resemblance. "What led to what" is a question about edges between specific things, and no amount of similarity search answers it, because two ideas can be causally linked and not resemble each other at all.

The build took a day, about eight hours. It started with NetworkX driven from LangChain, and moved to a hosted Neo4j instance when I found I wanted Cypher queries and better visualisations than the library gave me.

The pipeline is short enough to state completely:

raw entries
  -> extract the ideas in each entry
  -> embed each idea as a vector
  -> for pairs that sit close together, ask a model: does A influence B?
  -> keep the yeses as edges:  (idea A) -[INFLUENCES]-> (idea B)

One relation type, and that is deliberate. A richer schema of relations would need me to decide in advance what kinds of connection exist between ideas, which is exactly the thing I was building the graph to find out.

The similarity shortcut

Look again at the third line of that pipeline, the one that says "for pairs that sit close together". Nothing about the design requires that filter. The honest version of the question is asked of every pair: take each idea, walk it against every other idea, and ask the model whether one influenced the other. With a thousand ideas that is half a million model calls, and with two thousand it is two million, because the pairs grow as the square of how much I have written. I could not afford it, so I used distance to pick which pairs were worth asking about.

Which is the substitution from the retrieval section coming back to collect. Distance stands in for resemblance, and resemblance is a decent proxy for relatedness right up until the two ideas are causally linked and do not sound alike. A note about sleep and a note about a deployment schedule land nowhere near each other in the numbers, and one may well have caused the other. That pair is exactly the kind the graph was built to find, and the filter that made the graph affordable is the reason it will never be asked about.

I left it there, with the connections it did surface reading better than anything the raw queries had given me, and with that hole sitting in plain view.

Two years later the thing I run over the same corpus is different in almost every respect, and the one September conclusion that survived intact is the one about precomputation. Summaries generated ahead of time, embeddings stored, and the expensive model call saved for a small candidate set that cheap machinery has already narrowed.