How we built the CosmWasm simulator

Motivation

A smart contract is a program that holds funds and runs on thousands of machines at once, none of which you control. Every machine in the network runs the same program against the same inputs and has to arrive at the same answer, because the answer is what everyone agrees the balances now are.

That requirement makes the environment a strange place to develop in. A blockchain only appends, so nothing that happens can be taken back, and a machine that is busy agreeing with thousands of others is slow by construction. Development wants the opposite of both. You want to run the same broken thing forty times from a clean slate, change one line, and run it again, all in the time it takes to lose your train of thought.

The usual answer is a local fake. Ethereum developers had Ganache, which ran a private chain on your laptop with instant blocks and a reset button. On Terra, where contracts run on a system called CosmWasm, there was nothing equivalent. In May 2020, working on developer tooling at Terra, I wrote the whole ambition down as one line in a journal: modify Ganache to work with Terra. Two years later, at Terran One, I finally went at it properly, and the thing that came out was not the thing I set out to build.

The contract interface

A fake is only useful if it is indistinguishable from the real thing at the boundary the program can see, so the first question is what that boundary is made of.

A CosmWasm contract is compiled to WebAssembly, which is a portable instruction format designed to run inside a host that controls what the code can reach. WebAssembly on its own can do almost nothing. It cannot open a file, make a network call, or read a clock. Everything it can touch is handed to it by the host as a list of functions, called imports, and everything the host can ask of it is a list of functions it exposes, called exports.

For CosmWasm the imports are short enough to fit in a paragraph. Five of them are storage: db_read, db_write, db_remove, db_scan, and db_next. A few validate and convert addresses and verify signatures. One, query_chain, lets the contract ask a question about the wider chain. The exports are the entry points the host calls: instantiate, execute, query, and in later versions migrate, reply, and sudo. On each call the host also hands over the block height and time, the contract's own address, who sent the message, and what funds came with it.

There is one more property, and it is the one that decides the shape of everything else. A CosmWasm contract cannot make another contract act while it is still running. It can ask another contract a question and get an answer immediately, because questions change nothing, but to make something happen it returns a list of messages describing what should be done and then it stops. The chain carries those out afterwards. I had written the documentation for this at Terra two years earlier, and the sentence I used then was that contracts can only modify blockchain state through the chain's module message handlers, which prevents a contract from being re-entered partway through by something it set off.

Programmers have a name for handing control forward instead of holding it on a stack, which is continuation-passing style, and CosmWasm makes you write in it whether you want to or not. A program in that style has no partially finished work sitting anywhere. Between one message and the next there is no stack frame paused mid-function with local variables in it, because every unit of work ran to completion and left a list of what to do next. I did not notice that at the start, and it is the fact that decides what a simulator can be.

First design: a state debugger

On the second of August 2022 I wrote a line in my daily notes that said to write out the spec for a CosmWasm debugger and what it could be, and then I wrote it.

Debugger was the natural word, and it is worth seeing why, because the wrongness of it is not obvious from where I was standing. When a contract misbehaves the thing you want is to see its storage, and the tool that shows you a program's memory while you poke at it is a debugger. That is the whole reasoning, and it holds right up until you ask what "while" means for a program that is never partway through anything.

The spec is a debugger, straightforwardly. You upload a .wasm file, instantiate it, send it an execute or a query message, and watch the state change. It holds a history of states, a trace of which wasm calls fired, and a history of messages. The interface I drew has a list of contracts down the left, the current state in the middle, the difference against the previous state beside it, and along the bottom a timeline you can drag, with a marker at every state transition.

There was a version ladder attached, and it is the part of the document I still like: one contract first, then several, then a mock blockchain around them, then custom code hooks, then Rust integration, then editor integration. Each rung is a thing you could ship.

Two weeks later I added one line to a worklog about a sequence diagram view, which was a way to put multiple contracts in context with one another. That is a small note and it was the first crack in the design, because a view showing several contracts at once is not a debugger feature. Debuggers step through one program.

Continuation-passing style and the scheduler

The user interface came first. The cw-simulate-ui repository was created on the ninth of August 2022, and the cw-simulate repository it was a user interface for was not created until the second of September.

That ordering caused the problem I ran into, and the note where I worked it out is dated the second of September. What I wrote was that in order to communicate clearly what the tool was, I would have to spend time making a headless state management library, because otherwise the interface would have been spread all over the place. Then, having spent time looking at how WebAssembly debugging tools work, I found that what I had was not a state debugger at all. It was a JavaScript runtime for CosmWasm that ran in Node and in the browser.

The causal order inside that note is the interesting part, and it is the reverse of how design is usually described. Wanting to explain the tool forced the split into a core and a view. Making the split revealed what the core actually was. The name changed because the architecture changed, and the architecture changed because I tried to write down what the thing did.

Looking at it now, the deeper reason is in the contract interface itself. A CosmWasm contract hands control forward and stops, so there is no call stack to step through. What you have instead is a queue of messages waiting to be carried out, and something has to carry them out in order, carrying state from one to the next. A program written in continuation-passing style needs a scheduler, not a debugger. I had been trying to build an inspector for a thing whose defining feature is that there is nothing to inspect between steps.

The same note splits the work into four projects with one-line definitions: cwsimulate as the runtime with state management, cwsimulate-ui as the interface, cw-vm-js as the JavaScript runtime that executes functions inside contracts, and cwdb as a debugger in the style of GDB that would read debug information embedded in the wasm binary. It also fixed the layering, which had been muddled: the simulator is the chain, the JavaScript VM is the contract runtime, and the debugger sits on top of both. The repository was created the same day the note was written.

The app model: configuration and modules

The model underneath the simulator is a deliberate copy of the system it is imitating.

Cosmos chains, which is the family Terra belonged to, are assembled out of modules. A module owns some state, handles the messages addressed to it, and answers queries about itself. Bank is a module and it owns balances. Wasm is a module and it owns contract code and contract instances. A chain is a configuration plus a set of modules, and that is all a chain is.

So a simulated chain is a configuration plus a set of modules. My note from the time draws it in four words: an app has config and modules, and a module is a keeper, a message handler, and a query handler. The published package has a modules directory containing bank.ts, base.ts, and wasm, which is the same shape.

Copying the structure is what buys fidelity. If the simulator groups things the way the chain groups them, then a behaviour you observe in the simulator has a place it corresponds to in the real system, and you can go look. If the simulator invents its own arrangement, every difference between the two becomes a question you cannot answer without reading both.

The interface that came out of it is small enough to show whole:

import { CWSimulateApp } from '@terran-one/cw-simulate';

const app = new CWSimulateApp({ chainId: 'phoenix-1', bech32Prefix: 'terra' });

const codeId = app.wasm.create(sender, wasmBytecode);
let result = await app.wasm.instantiateContract(sender, funds, codeId, { count: 0 });
result = await app.wasm.executeContract(sender, funds, contractAddress, { increment: {} });
result = await app.wasm.query(contractAddress, { get_count: {} });

You create a chain, upload some bytecode to get a code id, instantiate it to get an address, and then send it messages. No node, no network, no Rust toolchain. Underneath, the store is a key-value store with prefix scoping and transactional wrappers, which is what makes the reset button real rather than a promise.

Instrumentation at the host boundary

The part I was actually chasing was never the simulation. It was being able to see inside a contract you did not write.

The obvious approach is to read the source. Contracts on CosmWasm are written in Rust, and Rust is a large, expressive language, which is the problem. I wrote the objection down plainly: the difficulty with analysing Rust is that there is an infinite range of possibility in how a developer can write a contract. You can find the storage calls in a simple contract by reading it. You cannot promise to find them in every contract, because there are always more ways to write the same thing than you have enumerated.

So the analysis moved. If you cannot constrain what the code says, instrument what it touches. Everything a contract remembers goes through those five storage functions, and the contract does not implement them, the host does. Replace the host's versions and you see every read and every write, from any contract, written in any language, without reading a line of it.

That idea is fifteen lines of TypeScript in the published package:

export class CWSimulateVMInstance extends VMInstance {
  constructor(public logs: Array<DebugLog>, backend: IBackend) {
    super(backend);
  }

  do_db_read(key: Region): Region {
    let result = super.do_db_read(key);
    this.logs.push({ type: 'call', fn: 'db_read', args: { key: key.str }, result: result.str });
    return result;
  }

  // the same override for do_db_write, do_db_remove, do_db_scan, do_db_next
}

Subclass the virtual machine, override the five storage imports, call the original, record what went past. The contract runs exactly as it would have and has no way to notice.

The larger version of this had a name, OverseerVM, and it was aimed at security auditors. It came in two halves. The first was an instrumented VM that customises behaviour by replacing the wasm imports implementing the CosmWasm API. The second was an instrumented backend with programmable versions of the three things a contract can reach, which are the API, the storage, and the querier. My note gives the example directly: attach predicates to storage that fire logging events when storage is touched. With that in place you can fuzz a contract and watch what it does to its own state, or state a property about storage and test it over many runs.

The move underneath is the one I keep coming back to in other work. When you cannot govern what something does, govern what it can reach, and put the instrument at the boundary rather than inside the thing being measured.

The mechanism and the product

The instrumentation inside the simulator did enough of cwdb's job that the separate tool stopped being urgent. cwdb was the GDB-style debugger reading symbols out of the wasm binary, defined in September, and the simulator absorbed the part of it people actually needed.

OverseerVM went the same way. The mechanism is the fifteen lines above and it runs; the auditor-facing product around it stayed a specification. The mechanism is the interesting half and the product is the useful one, which is a fair description of where the work stopped.

Some things were considered and turned down with a reason, which I still think is the right way to close a question. In January 2023 I looked at exposing the simulator as a server so several people could share a session and debug remotely, and wrote that it was already good locally and I could not name the benefit. Around the same time I floated compiling the simulator itself to WebAssembly so it could be driven from Python or Ruby, and floated going further and putting it inside the chain software directly, and neither got past the page.

One idea I still like is in a to-do list from November that never got ticked: take the trace format the simulator produces and add it to the chain's own code, so the real chain could emit what the simulator emits. That would have made the two systems say the same thing about the same execution, which is what you want from a simulator and a chain that are supposed to agree.

The published parts are on npm as @terran-one/cw-simulate and @terran-one/cosmwasm-vm-js, and the VM's own README carries the caveat that matters: great care was taken to match the behaviour of the original CosmWasm VM, and the results should still be checked against it for anything critical. A model of a system is a claim about that system, and the claim is only as good as the next time you check it.