Every model in this series reached some kind of resolution. A 9B model called the tool correctly and finished. A 4B model called it incorrectly and finished. Even a 2B model, on its worst runs, finished, sometimes with a wrong tool, once with a fabricated success message. An 800M model did something categorically different: it never finished at all. Generation would enter a tool-call assembly state and stay there, with no output, no error, and no natural stopping point, until it was manually cut off. That specific failure, not wrong, not slow, but structurally incomplete, is worth its own explanation.
Table of Contents
- What Actually Happens at 800M
- Why This Is Different From a Slow or Wrong Answer
- The Context Confusion Explanation
- Why the Tool Result Round Trip Is the Breaking Point
- How the Other Model Sizes Actually Compared
- What Would Actually Fix This
- What This Means for Choosing a Model Size
- What’s Next in This Series
- FAQ
- Related Articles
What Actually Happens at 800M
Running the standard test prompt from this series against an 800M parameter model, using the same mcp-email-server instance by wh1isper used throughout, produced the same outcome on every attempt. The interface would show a reasoning phase, then transition into an “assembling tool call” state, and remain there indefinitely. No malformed arguments appeared. No error surfaced. No partial tool call was ever submitted for execution. The only way to end the session was to manually stop generation.
This is a meaningfully different failure than anything documented elsewhere in this series. A model that submits a wrong tool call, or a correctly named tool with bad parameters, has at least completed the structural task of producing a tool call. An 800M model didn’t get that far.
Why This Is Different From a Slow or Wrong Answer
It’s worth being precise about what this isn’t. It isn’t simply a slower model taking longer to arrive at the same kind of answer a larger model gives quickly. Given enough time, a 2B model in this series eventually produced something, a wrong tool call, a hallucinated success, an honest “I am stuck.” An 800M model did not appear to be converging toward any of those outcomes given more time; it appeared to be stuck in a state that had no exit built into the reasoning process at all.
That distinction matters for anyone building around this. A slow-but-correct model is a latency problem, solvable with patience or better hardware. A model that enters a non-terminating state on a specific class of input is a reliability problem that no amount of waiting resolves.
The Context Confusion Explanation
The most useful explanation for this failure pattern, offered directly by Dr. Tim Frey while reviewing this testing, centers on what happens after a tool executes, not before. His framing: when a tool call completes, its result gets fed back into the model so it can process that result and continue the conversation. That return trip, tool result back into context, back into the model’s ongoing reasoning, is where a small model loses track of what’s already happened. For a model at the 800M scale, his assessment was that the model doesn’t have the capacity to hold the shape of that round trip together at all, and the reasoning process never resolves into a completed call.
This is a specific and testable claim, not a vague appeal to “the model is too small.” It says the problem isn’t raw context window size in the way that term usually gets marketed. A model can have a large advertised context window and still lack the capacity to track the specific back-and-forth structure a tool call requires, calling out, waiting, receiving a result, and correctly continuing from there.
Why the Tool Result Round Trip Is the Breaking Point
The Model Context Protocol is explicitly designed around exactly this round trip: a model requests a tool call, the host executes it, and the result is returned to the model to inform its next response. That’s the entire mechanism that makes tool use useful, without it, the model has no way to know whether an action succeeded, failed, or produced information it needs to continue the task.
The theory, based on what was observed here, is that this round trip is a distinct cognitive demand from the earlier steps in the pipeline, understanding the request, and identifying which tool applies. A model can be competent enough to get partway through, recognizing that an email needs to be sent, starting to construct a call, without having the capacity to also correctly model what happens after that call returns. At 800M, the testing here suggests the model doesn’t get past the construction step at all, which would be consistent with a model that, at some level, cannot resolve the shape of the full request-response cycle before it’s even attempted the call.
How the Other Model Sizes Actually Compared
It would be convenient, but inaccurate, to describe every model larger than 800M as simply working. The real picture is more specific, and worth stating precisely rather than rounding up.
| Model Size | Completed a Tool Call | Result |
|---|---|---|
| 800M | No, in any tested configuration | Never resolves, must be manually stopped |
| 2B | Yes, but inconsistently | One verified success; other runs produced a hallucinated success or a wrong-tool loop |
| 4B | Yes, consistently once fixed | Wrong parameters by default; correct and repeatable after shortening the tool description |
| 7B | Yes, inconsistently | Right tool most of the time; parameter accuracy varied across runs |
| 9B | Yes, consistently | Correct tool and parameters on every tested run, no changes needed |
Only 9B was reliable without any intervention. 4B became reliable after one specific, verifiable fix. 7B and 2B remain inconsistent even after the same fixes were applied. 800M is the only size in this series that never reached the point of producing a tool call at all, which is why it sits in a category of its own rather than simply at the bottom of a shared scale.
What Would Actually Fix This
Based on the context-confusion explanation, the fix isn’t a better prompt or a shorter tool description, both were tried elsewhere in this series and neither addresses a model’s fundamental capacity to track a request-response cycle. The proposed fix is architectural: render a tool’s result directly to the user rather than routing it back through the model for further processing and a generated response. That removes the exact round trip theorized to be the breaking point, at the cost of the model no longer being able to reason about or comment on the result itself.
That’s a platform-level change, not something adjustable through prompting, tool descriptions, or model configuration, and it wasn’t something testable within the scope of this series. It’s included here because it’s the most concrete, falsifiable explanation available for why 800M behaved so differently from every other size tested, rather than a hand-wave about model size in general.
What This Means for Choosing a Model Size
For a single-tool agentic task with a moderate parameter count, the practical guidance from this series holds: 9B was the only size that required zero additional engineering. Everything below that required either a specific, verifiable fix or came with residual inconsistency that would need to be planned around, not eliminated. 800M is the one size in this series that isn’t a reliability trade-off to manage, it’s a size at which the task doesn’t complete at all, and no configuration tested here changed that.
FAQ
Why does an 800M model fail to send an email through a tool call? In this testing, the model never completed constructing a tool call at all, unlike larger models that either succeeded or produced an identifiable wrong result. It entered a non-terminating generation state instead.
Is this a slow model problem or a capability problem? A capability problem. The model did not appear to be converging toward any outcome given more time; it remained in an incomplete state indefinitely rather than eventually producing a delayed but complete answer.
What is the context confusion problem? A proposed explanation, from direct review of this testing, that small models lose track of the request-response cycle inherent to tool calling, specifically the step where a tool’s result is fed back into the model for further processing, and that this round trip demands more capacity than earlier, simpler steps in the task.
Does a larger advertised context window fix this? Not necessarily. The theory here concerns the model’s capacity to track the structure of a tool call and its response, which is distinct from raw context window size as commonly marketed.
Did any prompt engineering fix the 800M failure? No. Simplified tool descriptions, explicit parameter naming, and directive system prompts were all tested elsewhere in this series against small models, and none addressed this specific failure mode for an 800M model.
What would actually fix this, based on this analysis? An architectural change where a tool’s result is rendered directly rather than routed back through the model for further reasoning, which would remove the specific round trip theorized to be the breaking point. This wasn’t testable through configuration alone.
Were all the larger models in this series fully reliable? No. Only a 9B model was reliable with no additional changes. A 4B model became reliable after a specific fix. 7B and 2B models remained inconsistent even after the same fixes were applied.
Is 800M viable for any agentic tool-calling task? Based on this testing, not for a task with a comparable number of required parameters to sending an email. Simpler tools with fewer parameters were not tested here and may behave differently.
How is this different from the 2B model’s failures documented elsewhere in this series? The 2B model did eventually produce a result on every run, sometimes correct, sometimes a wrong tool, once a fabricated success. The 800M model did not reach a completed result in any tested run.
Why does this matter for local AI deployments specifically? Because smaller models are often chosen for speed and lower hardware requirements. Knowing that a specific size threshold represents a structural failure rather than a manageable trade-off changes the calculation for whether the resource savings are worth it.