Will ChenA task list shared with an agentJulius designed the planner, and I connected its interactive task cards to the same data the assistant used.
Building Idyllic - Prior Iterations
January-February 2025
In January 2025, I wanted to plan my day with an assistant and then use the resulting plan as an ordinary task list. If I asked it to add a task, that task should appear in the planner. If I checked a task off myself, the assistant should work from the updated list the next time we spoke.
A checklist inside a chat response only solves the first part. It shows what the model suggested at the time it answered. Once I change the actual plan, the old response becomes a second version that I have to reconcile with the application. I wanted the conversation and the planner to refer to the same stored tasks.
The planner and its task data
Julius designed the planner around expandable task cards. A task could contain subtasks, and the cards sat inside named categories. The hierarchy let a large activity remain compact until I needed to work through its details. A checkbox gave me a direct way to change its status without having to describe the change in another message.
I implemented his components in Storybook first, which let me work on their appearance and interaction without connecting the database at the same time. The screenshot comes from that component test, so its example tasks demonstrate the interface before the later data integration. Once those components worked, I assembled the planner pages and connected them to Convex, the database we were using for live application state.
The page structure followed the task hierarchy. The planner opened on a day, a task had its own detail view, and a subtask could have a deeper view. Selecting a date also supplied context to the chat. Otherwise a request about “today's tasks” could refer to a different day from the one I was looking at.
I gave each application a small configuration record. The planner registered the components it could display inside chat and a context provider that supplied the selected date. This kept planner-specific behavior together instead of adding another special case to the main chat component whenever the application gained a feature.
Returning a component from a tool
Julius also proposed using AI in the add-task interaction. I implemented that alongside the planner, then connected interactive components to tool calls. A tool call lets the model ask application code to perform an operation. In this case, the response from that operation could include instructions for displaying a task component as well as the data the model needed for its answer.
My first attempt used structured model output to describe the interface, but that approach did not fit the tool-calling flow I had in place. I moved the interface description into the tool result. When a result contained a genUI field, the backend attached it to the chat response, and the frontend used the application's registry to display the corresponding component.
That arrangement gave the component several possible behaviors. A static result could show data captured at the time of the call. A live component could use an ID to query the stored task, and an interactive one could also send mutations back to the database. The live task card therefore had a way to stay consistent with the planner after the original chat message had finished.
The interface needed rules of its own. One request might cause several tool calls, each returning a component for the same task. Showing all of them produced duplicate controls inside a single answer. I added deduplication because the number of backend operations did not determine how many task cards a person needed to see.
Reusing objects outside the planner
By February, I was extending the same idea to objects beyond the planner. A named object could expose operations, and an operation could create another object that remained available to the conversation. The prototype below invokes a summary method and returns a reference to the resulting summary. The method description and object names stay visible, so I can see what the assistant is operating on.
I described the mechanics separately in Semantic objects: giving models handles instead of data. The planner supplied the complementary interface requirement: a reference also needed a useful view and, where appropriate, controls. A task deserved a checkbox. A collection needed a way to narrow it. A generated summary needed to remain addressable after the message that created it.
The planner’s task cards used their Convex query and mutation bindings inside completed chat messages. Finishing the model response did not disable the controls or freeze the data they displayed.