Why I gave the AI object references instead of text

In January 2025 I decided that tool calling was the wrong abstraction, and spent a month looking for a better one.

The problem only becomes visible once you know how an AI assistant actually works. A language model has no memory between requests. Everything it knows about your situation has to be written into the request itself, as text, every single time. That block of text has a size limit, and the model reads all of it before producing a single word of output. When people say an AI has a context window, that is what they mean: a fixed-size sheet of paper that has to hold the entire situation.

Tool calling is how a model reaches outside that sheet of paper. You describe some functions it is allowed to call, it writes out a call, your code runs it, and you paste the result back onto the sheet. That works well for one lookup. It works badly the moment you want several operations in a row, because every intermediate result has to come back through the text and be read again before the next step can be chosen. The model cannot hold anything. It can only look at things.

Motivation

I hit the wall on tool calling with my own journals. I had about two and a half thousand entries collected over Telegram, and I wanted to ask questions that ranged over all of them. There was no way to hand the model the collection. I could hand it entries, and the entries would eat the sheet.

Adjacent work: a JavaScript interpreter

The obvious answer is to stop passing data at all and pass a program instead. Give the model a JavaScript interpreter, let it write code against your data, and run the code. The data never enters the conversation, only the answer does.

I took that seriously and rejected it for two reasons, both of which are about the cost of the language rather than its power.

The first is that a general-purpose language is verbose about common operations. Filtering a collection by date range is a few characters of intent and a dozen lines of JavaScript, and every one of those characters is a token the model has to produce. If the same task can be expressed as one call, the cheap version wins on every request forever.

The second reason matters more. A language model does not write a program the way a compiler emits one. It samples each token with some probability of being wrong, so a longer program has more opportunities to go wrong, and a program that is wrong in the middle fails in ways that are hard to detect from the outside. My note at the time put both together: a JavaScript interpreter "takes a lot more tokens to do the same common tasks cuz JS is too low level. and low reliability due to stochastic nature."

I did not treat that as settled. The plan I wrote down was to prototype in JavaScript first, watch which operations kept recurring, and promote those into higher-level calls, because the argument above is an intuition and the recurring operations would be evidence. That is the honest version of the reasoning and it is the part I would keep if I could only keep one.

The object model

Work backwards from the constraint. The sheet is finite, so the collection cannot go on it. But the model is the thing deciding what to do with the collection, so it has to be able to name the collection in order to say what it wants done to it. Those two facts fit together exactly one way. The model gets something small that stands for something large, and the something large stays where it is.

That is a reference, and programming languages have had them for fifty years. I used classes from object-oriented programming as the guide, because the shape was already right. An object bundles some data with the operations that make sense on that data, and it has a name you can pass around without carrying its contents. That last property is the one I wanted. If the model can hold a name for two and a half thousand journal entries, it can talk about them without reading them.

The structure I wrote down was four levels deep and fits on a screen:

Structure for semantic object
  kind (should be like: so.idyllic.prototype#BlogPost )
  AI data - metadata so the AI knows if it's relevant
    description
  data fields
  methods fields
  data
    name
    type
    description
  methods
    name
    method signature
    description

A kind is a namespaced identifier, so the system can look up what sort of thing it is. The description under "AI data" is written for the model rather than for a person, so it can judge whether the object bears on the question at hand. Then the two lists: data fields with a name, a type and a description, and methods with a name, a signature and a description.

The description fields are doing more work than they look like they are doing. In an ordinary program, the compiler knows what a method does because it can read the method. Here the caller is a language model that will never see the implementation, so the description is the interface. Getting those sentences right is the same work as getting a function name right, except the audience reads English.

Underneath sat a small set of operations, which I thought of as system calls in the sense an operating system means it. The question I asked myself was "what type of system tools should our AI possess? they should be fundamental like system calls in an OS," and the answer I had at the time was five:

Only invokeMethod moves data. The other four are ways of finding out what exists and what can be done to it, which is the ratio you want when the caller is paying by the token to read anything.

Naming the identifier field

The smallest decision in the design took the longest, which is normal.

Every object needed a field naming what sort of thing it was. I could not call it id, because that was already the unique identifier of a particular object rather than of its type. I tried ref and did not like it. I tried URI and did not like that either, since the field held something more like a type name than a location.

The winner was kind, and the reason was about the reader rather than about elegance: "type is common in json so i want to say kind." A model has seen an enormous quantity of JSON in which type means a dozen incompatible things. Choosing a word it has seen less ambiguously is a real consideration when your caller learned the language by reading the internet, and it is the sort of consideration that did not exist five years ago.

Acceptance criteria

Before building anything I wrote down what would count as working. Seven lines, and the second one is the whole design:

• I can use them in prompts via the @-mention mechanism
• when the request hits the LLM, it doesn't resolve into text immediately
• the relevant context about the semantic object is injected (when necessary) before
• instructions for how to interpret the data
• instructions for what each method does
• there is data contained in the semantic object (data section)
• there are functions contained in the semantic object

"When the request hits the LLM, it doesn't resolve into text immediately" is the sentence everything else serves. If mentioning your journals in a prompt silently pastes your journals into the prompt, nothing has been built. The mention has to survive as a reference all the way into the model's view of the world.

Composition

I built the journal collection first, with about seven hundred entries loaded, and pointed Claude at it. The trace below is the running prototype, abridged only by cutting the JSON formatting.

It began by asking the object to describe itself, and got back an interface it had never been told about:

Tool call: analyzeObject   {"objId": "journal-collection"}

Tool result:
  This is a Journal Collection (so.idyllic.JournalCollection).
  A collection of journal entries

  Fields:
  - title
  - content
  - date

  Methods:
  - getDayCount: Returns the total number of days that have journal entries
      getDayCount() -> number
  - getEntriesForDay: Retrieves all journal entries for a specific date
      getEntriesForDay(date: YYYY-MM-DD) -> JournalEntry[]
  - getTotalEntryCount: Counts the total number of journal entries across all days
      getTotalEntryCount() -> number
  - getDateRangeView: Creates a filtered view of entries between two dates
      getDateRangeView(startDate: YYYY-MM-DD, endDate: YYYY-MM-DD) -> JournalCollection

Read the return type on that last method. getDateRangeView does not hand back entries, it hands back another JournalCollection. So the model called it for April:

Tool call: invokeMethod   {"objId": "journal-collection",
                           "methodName": "getDateRangeView",
                           "args": {"startDate": "2024-04-01", "endDate": "2024-04-30"}}

Tool result:
  New semantic object: 4f6b305b-3f7c-472a-9ac6-bc63b3045c8c - My journal collection Q2 2024
  (2024-04-01 to 2024-04-30) Date range view of A collection of journal entries from April to
  June 2024 with 28 days created. Please make another tool call to inspect or manipulate it
  further.

Tool call: invokeMethod   {"objId": "4f6b305b-3f7c-472a-9ac6-bc63b3045c8c",
                           "methodName": "getDayCount", "args": {}}

Tool result: 28

A new object came back with its own identifier and no entries in it at all. The model then called getDayCount on the narrower thing exactly as it would have on the wider one, and learned there were twenty-eight days in April, having read none of them. It did the same for June and got thirteen.

That is composition. Each step hands back a handle the next step can take, and the sheet of paper stays almost empty while real work happens behind it.

Three honest notes about that trace. The description string says "from April to June 2024" while the arguments say April, because the new view inherited its parent collection's description, which covered the quarter. The call that fetched the interface is analyzeObject, which is not one of the five I had written down a few hours earlier. And the object reports its kind as so.idyllic.JournalCollection, without the prototype# segment the structure spec calls for. The running prototype and the written design had already drifted apart in two places, which is what prototypes are for.

My log line from that afternoon reads "HOLY SHIT IT'S WORKING! Semantic Objects work with Claude," and I had checked off all seven criteria by the end of the day.

Adjacent work: retrieval

Anyone who works in the field asks this, and I wrote the answer down before anyone did.

Retrieval-augmented generation means searching a corpus for passages relevant to a question and pasting those passages into the prompt. My own note conceded the overlap and named the difference: "yes it is rag but dynamically programmable at prompt-time and easy to manipulate."

The distinction is that retrieval decides what you get before the model is involved, using a similarity search someone configured in advance. An object lets the model decide, at the moment it is reasoning, which slice it wants, then narrow again, then call a method on the result. Retrieval hands you a fixed sample of the collection. A reference hands you the collection.

What it cost

The idea kept going for another month, picking up a way for objects to announce changes to each other, and it acquired a name I used for pitching, which was object-oriented prompting.

In February I stopped it, and the reason I wrote down was about the name rather than the design: the narrative had started deciding what got built. What replaced it was narrower and more useful, because by April every object type I cared about had collapsed into a single one. Not a JournalCollection and a BlogPost and a HealthRecord, just a document, with the same handle property and none of the type zoo. That collapse is its own story.

What survived the whole year is the property the objects were built for. When an agent works with something large, the useful question is not what to put in front of the model. It is what the model can hold a name for.