I spend a lot of time thinking about how software actually runs on hardware. When you look at the current discussions around artificial intelligence, the focus is almost exclusively on scaling up. We see models with hundreds of billions of parameters, requiring clusters of GPUs just to load the weights into memory. But the edge, meaning the actual physical devices we interact with daily, does not look like a data center.
The edge consists of cheap microcontrollers, budget phones, wearables, and embedded systems. These devices operate under strict thermal constraints, limited battery life, and highly restricted memory. If you want to put an agentic model on a budget smartphone or a smart home hub, a general-purpose chatbot is the wrong architecture. You do not need a model that can write sonnets or explain quantum mechanics to turn on a living room light or set a timer.
This brings me to a recent release that caught my attention: Cactus Compute’s Needle 2 [1]. Cactus describes it as an open, Apache 2.0 licensed, 45-million parameter model built specifically for tool calling, device use, and structured extraction [1].
The interesting part of Needle 2 is not merely that it is small. It is that it accepts the constraints of its environment and optimizes entirely for a specific task profile. It trades general world knowledge for execution reliability in a narrow domain.
Why Task Scope Changes the Size Question
The conventional wisdom in machine learning is that smaller models are just less capable versions of large models. This is true if the evaluation criteria is open-ended chat, creative writing, or general knowledge retrieval. A 45M parameter model is not going to win a trivia contest against a frontier model.
But when we narrow the scope to device control and structured data extraction, the problem shape changes completely. If I ask a system to “dim the kitchen lights and play some jazz,” the model does not need to understand the cultural history of jazz or the physics of lightbulbs. It only needs to map my messy, unstructured human language onto a predefined schema of available functions.
Historically, we solved this with rigid voice command systems. You had to memorize the exact phrase, like “set living room lights to fifty percent.” If you deviated from the script, the brittle regular expression parsing failed. Tool-calling models fix this by acting as a flexible translation layer between natural language and an API. The model must select the correct function from a list, extract the relevant arguments from the user’s prompt, and format them correctly.
When framed this way, it becomes easier to see why a narrow tool-calling system may not need billions of parameters. All the capacity of the model can be dedicated to grounding arguments and selecting tools, rather than storing facts about the world. My architectural interpretation is that 45M is sufficient for this routing task, not a vendor guarantee about general capability. I see it as a routing problem, not an intelligence problem, because the application defines the available tools and expected schema.
The Mechanics of Constrained Execution
A tool-calling model is only as useful as its ability to adhere strictly to a contract. If a model hallucinates a function name or passes a string where an integer is expected, the entire execution pipeline crashes.
According to Cactus Compute, Needle 2 enforces part of this contract through a byte-level grammar compiled from the declared schemas [1]. The grammar constrains token sequences to syntactically valid JSON, but that does not guarantee the right tool name, sensible argument values, or a safe action. Schema validation and application logic still have to check semantic and tool correctness.
Consider a simplified example of how an interface might look on a smart device. The application defines a tool schema:
{
"name": "set_thermostat",
"description": "Adjust the target temperature of a specific room.",
"parameters": {
"type": "object",
"properties": {
"room": { "type": "string" },
"temperature_celsius": { "type": "number" }
},
"required": ["room", "temperature_celsius"]
}
}When a user says, “Make the bedroom a bit cooler, maybe 20 degrees,” the model has to populate that structure. The schema is the interface, but a valid JSON object can still contain the wrong room or an unsafe temperature. The same distinction applies to structured extraction from documents: the grammar handles syntax, while the model and application still have to get the meaning right.
Refusal and Bounded Confidence
One of the most critical features for an edge model is knowing its own limits. A small model will inevitably encounter requests it cannot handle. A generic chatbot might try to guess, hallucinating an action or providing a generic, unhelpful apology.
For a device control model, guessing is dangerous. You do not want your oven turning on because the model misunderstood a command about heating up a room. Cactus Compute says Needle 2 uses a confidence score for its responses, and that off-topic requests result in an empty call [1].
This refusal behavior is a feature, not a bug. It allows the local device to act as a fast, private filter. If the request is within the local model’s confidence threshold, it executes immediately. If the user asks something complex or out of scope, the local model refuses, and the system can safely fall back to a larger cloud model.
This architecture keeps routine control actions instant and free, while preserving the ability to handle complex queries via escalation. It acknowledges that the local model is not meant to be omniscient, but rather a reliable first line of defense.
Deployment Realities: Memory and Inference Budgets
When engineering for the edge, every byte matters. The memory hierarchy dictates performance. Reading from flash memory or DRAM consumes significant power and time, which is why shrinking the model size is non-negotiable for battery-powered devices.
The deployment figures in this section come from Cactus Compute’s published Needle 2 material, not measurements from this blog post. Cactus says Needle 2 is distributed as a single 14 MB binary and uses Cactus Quants for CQ2-bit compression, applied during training rather than post-hoc [1]. The vendor presents this as a way to fit the model into a small footprint; it is not an independent quality or speed result here.
The vendor reports a 28 MB peak session RAM figure [1]. Cactus attributes that footprint to co-design between the model and inference engine, including keeping the weights compressed and expanding 2-bit codes inside vector registers [1]. These are vendor-supplied implementation claims, not measurements I reproduced.
Furthermore, Cactus says attention uses a 256-token sliding window, which bounds the context window described in its material, and that the system prompt and tool declarations are pinned in memory [1].
The vendor reports 500 tokens per second of decode speed on a Raspberry Pi 5 [1]. It reports 400 to 1,500 tokens per second on VR devices such as Meta Quest 3S and Apple Vision Pro, and 300 to 700 tokens per second on sub-$200 phones such as the Samsung A-Series [1]. These are vendor-reported figures for separate device classes, not one directly comparable range. The same vendor page says Needle 2 runs on newer microcontrollers such as the ESP32-S3 with the reported memory budget [1].
Cactus also compares Needle 2 with FunctionGemma 270M, LFM2.5 230M, and Apple FM on its device-use benchmarks. The reported comparison says Needle 2 is 5x to 70x smaller than those baselines, with tradeoffs in accuracy and memory footprint [1]. Those are the vendor’s benchmark results, not an independent comparison in this post.
Building Systems That Fit
There is a specific kind of engineering satisfaction in building systems that fit their physical constraints perfectly. A massive cloud model is impressive in its capability, but it is often a blunt instrument for simple tasks. We do not need a supercomputer to parse a request to lock the front door.
Deploying a model like Needle 2 represents a shift in how we approach edge computing. We are moving away from trying to compress a general-purpose brain into a tiny box, and instead building highly specialized circuits for language translation and routing.
By focusing on typed tool schemas, structured extraction, bounded memory, and refusal boundaries, we can build narrower interfaces for physical devices. For this class of edge software, the goal is not simply to make a massive model smaller. It is to make a small model useful for a defined set of tasks.