I think I’m the target demographic for coding agents that run for hours on their own. I want one to just do a good job. Only ask me what it needs to ask me, and don’t ask me anything else.
That’s tricky, and both failure modes frustrate me. If it skips a question I cared about, it made a judgment call I wanted to own. And on a multi-hour run, a question I didn’t need to answer costs a lot, because I can come back to find it stopped and waited on me for the last two hours.
I think the interview is the right interface between a person and an agent. Deciding what an agent should ask is a tricky problem. I think there are technical answers for how questions get conveyed, but I also think it’s as gnarly as memory. And I don’t know how to end the interview.
Moving Questions Upstream
A blocking question in the middle of a multi-hour run can leave the agent idle until the human is back. One proposal is to move clarification as far upstream as possible: settle the acceptance criteria and the checks before the run starts, and let anything still unresolved keep asking on the ticket or in chat instead of blocking the run.
That only helps if the upstream conversation asks the subtle questions, the ones that affect architecture. If you miss them there, I don’t know how you get answers to them later. We’ve all come across features where one little UI change has huge downstream effects.
The Bulk-Run Example
One request I had was for bulk runs of something. From a UI standpoint that’s tiny: the input box takes multiple IDs instead of one, a multi-select instead of a single select. But the work is processed asynchronously. With one item, the status is success, still running, or failure. With many, you get partial failures: most succeeded, one is still running, one errored, and you only have one status field. What’s the composite status? Knowing when the whole batch has terminated, in an asynchronous system, is very difficult to do.
Now imagine that request goes through a product owner to an architect to an engineer, and the engineer hands it to an agent while wondering what to do. The agent will absolutely have an answer. Whether that answer is correct isn’t something I think you can tell at that point in time.
Memory and Better Tickets
There’s a tool that’s described as memory, but technically it remembers how people review PRs on a repo and feeds that upstream into the agent. If you keep having to complain about the same stuff in PRs, it remembers that. One proposal is to seed that memory from past PRs. My first concern is that past PRs may already be polluted by AI-generated reviews. How many of those reviews were written by humans? But I think it could still make sense, because that’s how you build properly: you take an inventory of what reviews look like and learn how to build on that repo.
Tickets are the other half. I don’t want them to get crazy structured, where you shove metadata into every ticket and people have to write them exactly one way. The dream is that you have a ticket, and if anything isn’t clear, it gets resolved in as few iterations as possible so the agent can run it. Shared skills for writing implementable tickets, and grill-style tools that interview you about a ticket before it’s handed off, can help make tickets better up front.
Where to Stop Asking
The harder part is how much the interview should ask. There’s a spectrum. On one end is a form with ten fixed questions: who is your user, what’s the expected user flow, what scale do you need to support. Fill it in and that’s your PRD. That’s not AI at all, because we already have those questions. On the other end is a free-form interview where the agent asks one question at a time based on all your previous answers, a choose-your-own-adventure that ideally works from the highest-consequence questions down to some threshold. How many questions will you answer? Unknown. It keeps going until you reach some conceptual agreement.
I think for this to be collaborative and productive it has to be somewhere in the middle, and I don’t know where that level is. The more questions you ask, the less likely people are to keep thinking about them. But a question you don’t ask hands the model a judgment call nobody made explicitly, and that call can get shipped. I could write into a skill “ask questions until you have a high-level architecture and all the functional requirements,” but then I’m relinquishing judgment to the model. What do all functional requirements look like to the model? It could ask a hundred questions on this edge case or that one, or ask one question and decide that’s good enough. I don’t have a termination rule, short of putting a hardcoded number of questions on it.
Execution is collapsing into a template, and I think that’s okay: once we’ve figured out what we want to build and how, it can be done in a templatized way. Planning isn’t there, because there are too many angles that different people look at and it’s more stylistic, and for now I think that diversity of approaches helps create richer products. So I’m not sure we’re ready to template planning yet, even the interview.