wcdcOS: designing a personal platform as an operating system

Motivation

I wanted my bank transactions annotated with what I had actually bought. The information exists, in the sense that the transaction is in one place and the receipt is in an email in another place, and nothing puts them together. So I wrote down what it would take, and what it took turned into an operating system.

The worked example

The worked example in my specification is an accounting application made of three scripts.

The first grabs transactions from several banks and inserts the new ones into a document store. The second grabs emails and inserts the new ones. The third takes a single transaction, searches the emails for anything relevant to it, reads them including attachments, and writes a summary back onto the transaction.

Writing the three scripts is easy. Getting them to run in the right order is where the specification starts, and here is what it had to say about it:

"1. sync-bank-data creates the following jobs:

1. schedule sync-email-data and take note of its job ID
2. for each NEW transaction added to database, create a annotate-expenses job with the ID of the transaction in the database.
3. scheduler runs sync-email-data in parallel, and upon completion updates it.
4. after sync-email-data is done, the scheduler runs annotate-expenses in batches."

One script has to be able to schedule another and hold onto a reference to that scheduled work. One runs once per item rather than once. Something has to know that the third cannot start until the second has finished. Nothing here is exotic, and all of it together is a scheduler, a queue, and identifiers for work that has been requested but not yet done.

Once you have written those three things down you have started an operating system, whether or not that was the plan. So I decided to write one on purpose, on the theory that a thing designed as what it is tends to come out better than a thing that becomes it by accident.

Kernel and userland

The first structural decision was to split the system the way a real operating system splits:

"wcdcOS is divided into Kernel and User contexts. The Kernel is the software layer that implements the internal plumbing of the system, like storing data or managing the various caches and queues. These concerns are carefully scoped and encapsulated in order to expose clean abstractions and interfaces to User-mode objects in wcdcOS."

The reason this is worth doing, and not just worth saying, is that it forces a decision about every capability: does a script I write on a Tuesday get to touch this directly, or does it have to ask? Writing out what lived in the kernel took five groups:

storage layer          realtime document database
                       virtual file system
                       resource abstraction
                       kernelmode and usermode cache

process mechanism      jobs queue
                       worker processes and pipelines
                       inter-process communication
                       scheduler

security               secrets management
                       authentication
                       permissions

messaging and event    realtime pubsub
system                 channels

audit system           logs

Nothing in that list is unusual for an operating system, which is the point. Once the accounting example forced me to admit I needed a scheduler and a queue, the rest arrived as consequences: a queue implies durable storage, durable storage implies scopes, scopes imply permissions, and permissions imply an audit log to tell you when one was used.

The permission system was borrowed openly, "similar to POSIX operating systems," based on users and groups. I sketched what the specification calls "a basic set of permissions" and got four of them down before trailing off into an "etc.," which is worth showing as written rather than tidied into a closed list:

The "etc." is the honest part. I knew the shape of the axis, which is that permissions are granted over kernel capabilities rather than over individual pieces of data, and I had not finished enumerating it. Sketching the axis is the load-bearing move and completing the list is bookkeeping that only real applications can settle, because you find out which capability you actually needed to separate the first time two scripts want different amounts of the same thing.

What makes the permissions concrete rather than decorative is how storage is scoped. The document database is cut up before anything is written to it:

kernel
userland
  app
    app's own virtual doc db
    process spawned by app
      process's own virtual doc db
  pipeline (spans multiple apps)
    pipeline's own virtual doc db

Every application gets its own database, every process an application spawns gets its own, and a pipeline, which spans several applications, gets one local to the pipeline. So "this script may read that data" is a statement about which scope it is running in, which is a thing the system can check, rather than a convention I have to remember.

The pipeline scope is the one that had to be argued for. Two applications cooperating need somewhere to put shared intermediate state, and the two available answers are to let one reach into the other's database or to give the collaboration a database of its own. The first is easier and quietly makes every pair of applications into one application. The second costs a concept and keeps them separable.

Applications, processes, pipelines, and jobs

Four nouns carry the design, and each is defined against the others:

"- an application is a user-land package of code which contains multiple pieces of functionality

• a process is an instance of code execution created by an application
• a pipeline is an instance of a process flow that can involve multiple applications and processes
• a job is a description of a requested task published to a queue to be assigned by the job scheduler"

The distinction that does the most work is between a job and a process. A job is a description of work, which means it is data: it can be written down, put in a queue, held until a condition is met, and handed to whichever worker is free. A process is an instance of execution, which means it is a thing happening, and it can be watched, timed, and found in a log afterwards. Keeping them separate is what lets the accounting example work at all, because "make one of these for every new transaction" produces descriptions, and something else decides when each becomes an execution.

The process message interface

Processes here are not machine processes, so the borrowed idea needed adjusting:

"A process can read/write data to other processes via messages. Processes can publish messages and subscribe to messages. Messages exist on a global message bus, and processes should define how messages should be formatted / structured in order to interact with them. This is called the PMI (Process Message Interface)."

The clause that matters is that processes define how messages should be formatted in order to interact with them. The bus itself carries anything and enforces nothing. Each process publishes its own contract for how to be addressed, and anything that wants to talk to it conforms to that.

The alternative would be a central schema that every message has to satisfy, and the cost of that alternative shows up later rather than immediately: every new kind of process becomes an edit to a shared definition that everything else already depends on. Letting each process own its own interface means adding one changes nothing that already works.

Bloom filters

One passage in the specification ends in a conditional rather than in a decision, which is where honest reasoning about an optimization usually ends up, and it is the part I still like most.

The problem is ordinary. I have a collection with a large number of items, each with an ID, and before inserting something I want to know whether that ID is already there. Loading the entire collection into memory to check is exactly what I am trying to avoid.

A bloom filter is a small structure that can answer a version of that question cheaply. It is an array of bits plus a handful of hash functions. To record an item, you hash it several ways and set the bit at each resulting position. To ask about an item, you hash it the same several ways and look at those bits. If any of them is zero, that item was definitely never added, because adding it would have set that bit. If all of them are one, the item might have been added, or those bits might have been set by other items that happened to collide. So the structure answers "maybe" or "definitely not," and never "definitely yes."

That asymmetry is the useful part, and it is easier to see laid out than described. My note at the time says "I need to clarify with a table," so I drew one of which combinations can actually occur:

One cell of four is impossible, and that impossibility is the whole value of the structure. There are no false negatives. When it says definitely not, you can skip the expensive lookup entirely and be certain you were right. When it says maybe, you have learned nothing and have to go and check.

From which the conclusion follows, and it is a conditional one:

"It seems that a bloom filter makes most sense as an intermediate probabilistic shortcut before a moderately expensive lookup; in this contrived scenario, the stakes are high as we would pull the entire collection if we get a "maybe". this optimization thus depends on how often "definitely not" gets reported instead of "maybe" when absent."

The optimization is worth having exactly when the cheap answer arrives often enough to pay for the times it does not, and how often that is depends on the size of the bit array against the number of items, which is a thing I would only learn by running it. There is also a constraint that rules it out for most collections: a simple bloom filter cannot remove elements, so this only works if the collection is append only.

That is what designing actually feels like from the inside. Most of the work goes into establishing the conditions under which the clever structure would be worth its complexity, rather than into picking the structure, and then into being willing to stop at "it depends, and here is what it depends on" instead of forcing a decision the evidence does not support yet.

The implicit programming model

At the end of the specification I noticed something about what I had written:

"This means that there is an implicit abstract programming model; as if there were a "language" behind the scenes as well."

Having defined applications, processes, pipelines and jobs, with scopes and messages between them, I had described evaluation rules for a system nobody had built. And the last idea in the document is the one I would still like to see:

"An idea: expose programmatically the "job" model to be like a Promise; it'll be like a distributed machine — imagine a JS event loop, but existing on a higher plane and not limited to the scope of a program execution on the machine!"

A Promise is the thing a program hands you when work has been requested and not yet finished, which is exactly what a job is. If jobs were Promises, then scheduling would be composition, and waiting for three overnight tasks would look like waiting for three network calls. The whole system would read as one event loop whose scheduled work happens to take hours instead of milliseconds.

Recognising the runtime

The pieces turned up separately, from outside, and having designed the kernel is what let me recognise them when they did:

"You needed automation which I got from Home Assistant. You needed N8n which had the visual builder and platform and the baseline for the OS. I needed the insight from WCDC OS which was going to be the agent environment which I was going to add all these agents so and I was already in the mindset of like having a bunch of agents doing my tasks … so when it all came together when I discovered N8n, this automation workflow thing, I realized that this could be put together and create the system that I was always imagining."

Three components, each from a different place. Home automation software supplied the idea that events in the world can trigger code. A workflow tool supplied the visual builder and, in my own words at the time, the baseline for the OS. My specification supplied the agent environment, which was the part neither tool had.

The design was not wasted by not being built. Written down, it became the thing that made an ordinary workflow tool legible as a kernel I already had a use for. I would not have recognized it otherwise, and I think that is the ordinary way most architecture work pays off: not as the system you build, but as the reason you know what you are looking at when you meet it.

Though I should not tidy this up too much, because the record does not. That recognition happened at the end of March, and in the middle of June I was still writing that for this month at least I wanted to be working on creating my personal wcdcos system. Finding the runtime did not stop me wanting to build the thing. It just meant I finally knew what the thing was made of.