Will ChenA visual builder for AI appsThe November app builder connected screens to executable flows, while inline semantic objects and December context experiments made results reusable.
Building Idyllic - Prior Iterations
November to December 2024
In November 2024, a button in Clarity Bear carried a surprising amount of responsibility. It had to take the contents of a document editor, send them through an AI step, store the result and move the person to the interview page. The next page then needed that result to begin its own conversation.
I could write those connections directly in application code. To make them editable in Idyllic, I needed a representation that connected what appeared on screen to the program running behind it. A diagram from November 8 puts the two sides next to each other: UI on the left, logic on the right.
In the drawing, a button click starts the outline-generation flow. A message in the interview triggers evaluation of the conversation and generation of another question. The result updates the clarity indicator. The review page receives a generated document. This made the relationship between a user action and an AI operation explicit enough to implement.
Connecting screens to operations
I began translating the hardcoded flows into an abstract syntax tree, or AST: a structured description of the operations for an interpreter to run. The visual editor supplied nodes and connections, and a parser turned those into the executable description. This let me work on the editor and the runtime as separate parts of the same system.
During November, I got an interpreter running and connected button interactions, navigation and string-based chat callbacks. A text element could receive streamed text from an AI node. Streaming also required more careful state updates; I fixed a race condition by partitioning updates by key, while streaming callbacks and initial state still needed work.
The graph itself was another design problem. An execution sequence tells you what happens next. When connecting AI calls and interface state, you also need to see where each value comes from. On November 20, I planned to move the editor toward data flow and revise how function results bound to state and appeared in the UI.
This capture still shows an action sequence: an entry point, an AI call, a state update and navigation to another page. It records the editor during that revision. I also needed to design expressions and type checking, because the way values moved between nodes would determine how understandable the graph could become.
Objects with their own interfaces
Alongside the app builder, I was exploring what kinds of things the interface could contain. On November 5, we described a “semantic object” as something with data, relationships, methods and a renderer or editor. In this version of the idea, an object carried enough structure for both an AI program and a person to work with it.
A color palette was a useful example. Its colors could be represented as data, but the person should see swatches and controls. A musical pattern should have a playable interface. The object's type would tell Idyllic how to display it and which interactions to offer.
The November prototype placed a palette, a small chart and a musical pattern inline. The still shows the visual objects and their controls. The broader proposal also included objects such as calendars, task lists and schedules, with relationships and operations between them. Those examples described the direction of the system; the visible prototype established the narrower question of rendering different object types together.
This required a way to choose and configure components. A related Figma design shows a library containing elements such as a button, chat, progress bar and an agent activity display. The activity preview distinguishes completed, active and pending steps. A component had to expose properties the builder could edit and events the runtime could connect.
Context inside the project
By December, I was using a chat application I had built and considering how to combine it with the builder. The December 10 proposal treated a project as a place with its own context, state, interface and logic. A person could work inside it immediately, then customize the parts they needed.
Context became a more specific design problem within that workspace. Joining strings into a prompt obscured the operations I wanted to perform on the information itself. I wanted to select material, add it to context, transform it or remember a result for later. On December 21, I was sketching an interface for those operations, with prompt templates kept separate from the contexts they would use.
Some of this was becoming ordinary application behavior. Agent mode could show a sequence of thinking steps. By December 25, I had added a project memory mode that searched other chats in the same project and included results in a new query. I still needed to make that retrieval more specific.
That left a concrete design boundary: storing information in a project was only the beginning. An operation needed to select the relevant parts, pass them to the model in the right form and put its result somewhere the person could use again. The screens, program representation and context model all had to agree about what that result was.