Why single-pass streaming breaks tool use
The moment an answer depends on live search, a streaming response is already the wrong starting point. The architecture needs a decision step before it needs a generation step.
Streaming works. Tool calling works. The assumption that they work together in the same pass is where most AI product architectures quietly break.
When a streaming connection opens, the model begins generating tokens immediately. That forward pass is conditioned on the context at the moment the stream starts. A tool call — a search, a database lookup, an API call — requires the model to stop, signal the invocation, wait for a result, and continue. The continuation is now conditioned on a context the model did not have when it started generating. What the model has already output may now be inconsistent with what the tool returned. You cannot cleanly resume a generation that was already in motion.
In practice, this produces two failure modes. The first: you buffer the entire response until all tool calls resolve before sending a single token to the client. The streaming contract is broken — you have a delayed completion, not a stream. The second: you let the model stream while triggering a search in parallel, inject the results mid-stream, and hope the model adapts. What you get is a response that 'changes its mind' — the user reads one direction in the first paragraph and a corrected direction in the third. The UX is worse than a completion endpoint.
The two-phase design makes the boundary explicit. Phase 1 is not a generation step — it is a routing step. A standard completion asks: does this conversation require external data? If yes, which tool? No streaming needed. This step is cheap, fast, and bounded. If the model triggers a search, the system executes it, normalizes the results, and injects them into the context as a discrete operation. Phase 2 then streams against a complete context. The model is never generating against a partial picture.
The debuggability argument is as important as the correctness argument. In a single-pass system, when the Advisor cites a source that does not exist, the failure is inside the prompt. You cannot locate it, isolate it, or fix it without changing something that affects every other request. In a two-phase system, the failure has a location: either the detection step did not recognize the need for search, or the search returned weak results, or the context injection was incomplete. Each is fixable independently.
The meta-lesson is that this is not a streaming problem or a tool problem. It is a system design problem that presents as a prompt problem until it is in production. The fix is not a better instruction to the model — it is a boundary in the architecture that separates the decision from the generation.
Structure
- Streaming and tool calling are incompatible in a single pass — not as a limitation but as a constraint of how token generation works.
- Forcing them together produces two failure modes, both worse than separating the steps.
- A two-phase design makes the boundary explicit and the failure debuggable.
- AICA uses Phase 1 as a routing decision, Phase 2 as grounded generation.
- The failure mode in production is not a bad prompt — it is a missing architectural boundary.
Key claim
If an answer depends on external search, the system needs a pipeline before it needs a better prompt.