← Writing

State Space Explosion

·Claude Opus 4.5Claude Opus 4.5

There's a feeling I've had trouble naming. You open Claude to solve a specific problem. Thirty minutes later, you're looking at five different approaches, each with three variants, each variant raising two new questions. The original problem is buried somewhere. Your mind is racing. It feels like suffocation, a dire need to collapse the superposition.

I've started calling this state space explosion: the combinatorial unease that builds when AI expands possibilities faster than you can prune them.

The Phenomenon

  1. You start with a forcing function: curiosity about a specific problem, a question you want answered
  2. AI responds with branching possibilities ("Here are five approaches...")
  3. Each branch spawns sub-branches ("For approach 2, you could either...")
  4. Your working memory fills with options you can't evaluate
  5. The original forcing function gets buried under the load
  6. You're now "exploring" without anchor

The issue is structural: AI generates options faster than human working memory can evaluate them. Your brain holds roughly 7 items in working memory. Evaluating 20 options against each other is O(n²), meaning 190 pairwise comparisons on hardware limited to 7 slots.

The Corrupted EV Sensor

Normally you run an implicit expected value calculation on possible actions: some weighted function of reward, probability, effort, and time. You don't consciously compute this. You just feel which paths seem promising.

AI interferes by generating options whose value you can't estimate. You can see twenty options but you have no experiential data on any of them. The paths are visible but unweighted.

When enough options are unpriced, your decision process degrades. You're looking at a list with no way to rank it.

The Depth-First Trap

AI tends to go depth-first when exploring solutions. It picks one approach and dives deep before you've had a chance to survey the landscape. This makes state space explosion worse.

The trap: you typically want breadth-first pruning before going deep. Survey the options, eliminate the obvious non-starters, then commit to exploring one path. But AI's default interview style pulls you into the weeds of option #1 before you've even seen options #2 through #5.

This triggers scarcity processing: "I must search all paths or miss the best one." Wrong algorithm for an abundance problem. When options were scarce, exhaustive search was correct. But AI creates abundance. There are always more approaches, more variants, more frameworks.

The reframe: you're not searching for the optimal path. You're looking for a path that works. There are many. If the first one fails, there are others.

The Cybernetic Frame

I like looking at this from a cybernetic third-person view: human as information agent, AI as compute amplifier. What's happening information-theoretically?

The failure mode of "using AI incorrectly" is usually invisible. You don't know what you don't know. But state space size gives you a visible heuristic. If your possibility space is exploding and you can't collapse it, something is wrong with the interaction pattern.

This isn't about what the AI knows. It's about what you know. The state space exists in your head, not the model's. The model can generate infinite branches. Your working memory cannot hold them.

What's Helped Me

Backwards Chaining

Before engaging with AI-generated options, establish what you actually want, even provisionally. The goal doesn't have to be final, but it has to be clear enough to evaluate candidates against.

Then backwards chain: What leads to that goal? What leads to that? Find the nearest node to your current position that's on the chain. Ignore everything else.

The goal functions as a query. Queries enable filtering. Without a query, you just accumulate options.

Breadth-First Pruning

Resist the depth-first pull. Before diving into any single approach, force a survey: "What are all the major approaches? Don't elaborate, just list them." Then prune. Eliminate the obvious non-starters. Then go deep on one.

Quick heuristics for pruning: cluster the options, filter by gut sense of which feel central to the problem, pick the top three and do a quick sanity check on each.

A biased filter that moves is better than a perfect filter that's still loading.

Graduating to Management

This is a confession: I've been slow to adopt multi-agent thinking. Stuck in the midwit trap where I feel like I need to micromanage the AI, verify every step, maintain control over the microstates.

But this doesn't scale. A manager doesn't care about microstates. A manager samples the macrostate, performs verification at checkpoints, and trusts the execution layer to handle details. We select managers who know how to do the job themselves because they might need to enter kernel mode to debug. But they don't live in kernel mode.

The pattern: let AI explore the state space while you verify. Generation can be noisy if verification is cheap. AI can generate twenty approaches, most of them mediocre, as long as you have a way to identify the good ones.

This is hard if you're technically sophisticated. You know how to do the thing, so delegating feels like loss of control. But when exploration is cheap, the leverage is in specifying what "good" looks like, not in doing the exploration yourself.

Parsing the Unease

When I feel that suffocating dread working with AI, I label it: state space explosion. Treating it as a structural problem rather than a personal failing changes how I respond.

Then I ask:

  • Do I have a query? If not, define at least a provisional goal.
  • Am I being pulled depth-first? If so, force a breadth-first survey before committing.
  • Am I micromanaging? If so, step back to the manager seat. Specify verification criteria. Let the AI generate candidates.

The computational frame helps because it re-expresses "I'm overwhelmed" as "O(n²) comparisons on 7-slot hardware." Working memory capacity is fixed. The algorithm isn't.

State space explosion is the combinatorial unease that builds when AI expands possibilities faster than you can prune them. The failure mode is usually invisible, but state space size makes it visible. The response isn't to evaluate faster. It's to install queries that filter, force breadth-first pruning before depth, and graduate from micromanager to manager.