14 MB On-Device LLM: What Cactus Needle 2 Changes
Sharan Sifat14 min read8 viewsCactus released a 45M-parameter agentic model that ships as a 14MB binary and runs on a Raspberry Pi at 500 tokens a second. Here is what it genuinely does, where early testers broke it, and the production pattern that makes it useful.
On this page
Cactus has released Needle 2, a 45-million-parameter on-device LLM that ships as a single 14MB binary and needs about 28MB of RAM to run a full session. It works on a Raspberry Pi 5, on sub-$200 Android phones, and on ESP32-S3 microcontrollers, with no GPU and no network connection. It is Apache 2.0 licensed.
The important thing to understand before you get excited is what it is for. Needle 2 is not a small ChatGPT. It does not write, reason at length, or hold a conversation. It turns a short natural-language instruction into a function call, or pulls structured fields out of a piece of text. That narrowness is exactly why it fits in 14MB.
We build AI features into commercial products, so the question we care about is not whether the benchmarks are impressive. It is whether you can put this in front of real users without it embarrassing you. The honest answer is: sometimes, in a specific architecture, with a fallback. This article covers what it does, where early testers broke it within hours, and the pattern that makes it usable anyway.
- Needle 2 is a 45M-parameter model in a 14MB binary, running in roughly 28MB of RAM with no GPU required.
- It is a dispatcher, not a reasoner. It maps instructions to tool calls and extracts structured data.
- On the Mobile Actions benchmark it scores 63.7%, below LFM2.5 230M's 69.1%. The win is size, not raw accuracy.
- Public testers broke it quickly on relative instructions like "make it a little warmer in here".
- The usable pattern is local-first with a confidence threshold that escalates to a bigger model when unsure.
What Needle 2 actually is
The numbers
Cactus publishes specific figures, and they are worth reading carefully because the size claims and the speed claims are both unusual.
- 14MBsingle binary, model included
- 28MBRAM for a full session
- 45Mparameters
- 70 MFLOPsper token
- 500 tok/sdecode on a Raspberry Pi 5
On decode speed, Cactus reports 500 tokens per second on a Raspberry Pi 5 with 800+ tokens per second prefill, 400 to 1,500 tokens per second on VR hardware like the Meta Quest 3S and Apple Vision Pro, and 300 to 700 on inexpensive Samsung A-Series phones.
The compute figure is the one that explains the rest. Needle 2 uses roughly 70 MFLOPs per token. Cactus compares that to 460 for LFM2.5 230M, 540 for FunctionGemma 270M and around 6,000 for Apple's foundation model. Running an order of magnitude fewer operations per token is what lets it hit those speeds on hardware with no accelerator at all.
Why 14MB is the whole story
A 14MB binary is small enough to bundle inside a mobile app without anyone noticing. It is small enough to flash onto a microcontroller. It is small enough that you stop thinking about model distribution as a problem.
That changes the deployment question completely. Most on-device AI conversations until now have been about whether a phone has enough RAM, whether the download is acceptable over cellular, and whether you need a separate model file that can go missing. Needle 2 sidesteps all three. The model, tokenizer and grammar compiler are sealed into one dependency-free C++ binary.
The achievement is not that a 45M model can call tools. It is that a model this small stopped needing a GPU, a network, or a download screen.
What "agentic" means here, and what it does not
A dispatcher, not a reasoner
The word "agentic" is doing heavy lifting in the marketing of every model released this year, so it is worth being concrete. Needle 2 does three things: it picks a tool from a list, fills in that tool's arguments, and extracts structured fields from text. That is it.
Think of it as a switchboard operator rather than an assistant. Someone says a short phrase, and the operator's job is to plug the call into the right socket with the right parameters. The operator does not need to understand the conversation that follows, form opinions, or remember yesterday.
Practically, in the Python package you decorate functions with @needle.tool, instantiate an agent, and call it with a query. The runtime retrieves a shortlist of candidate tools, picks one, and returns a confidence score alongside the call. Memory is deliberately bounded at a 256-token sliding window.
The architecture tricks that got it this small
Three choices do most of the work, and they are interesting because each one trades generality for size.
A Hadamard MLP replaces the usual dense projections with a fixed Walsh transform, which removes a large chunk of the weights you would otherwise have to store and read. An engram system moves world knowledge out of the network weights and into hashed n-gram tables, which is cheaper than teaching the parameters to memorise facts. Multi-lane residual streams give routing flexibility across what Cactus describes as a 27-layer, 512-wide network.
The quantization approach matters more than any of them. Cactus applies its CQ2-bit scheme from pretraining onward, covering weights, activations and the KV cache, rather than compressing a finished model afterwards. Because the model is trained at 2-bit precision from the start, the deployed model is the trained model. Conventional post-hoc 2-bit quantization tends to collapse the quality of small models, and this avoids that failure mode by never creating the gap in the first place.
Needle 2 is Apache 2.0 licensed, which is genuinely permissive. You can use it commercially, modify it and ship it inside a closed product without a revenue-sharing obligation. That is not a given any more, as we covered when Alibaba attached revenue terms to a Qwen release in our piece on open-source AI licensing. Check the licence yourself before you build on any open-weight model.
How good is it, honestly?
What the benchmarks actually say
Cactus's headline framing is that Needle 2 "trades wins" with models 5 to 70 times larger. That is a fair description, but the underlying numbers are more nuanced than most coverage has reported.
On the Mobile Actions benchmark, across 961 rows, Cactus's own published results are:
| Model | Mobile Actions accuracy | MFLOPs per token | Relative size |
|---|---|---|---|
| LFM2.5 230M | 69.1% | 460 | ~5x larger |
| FunctionGemma 270M | 64.0% | 540 | ~6x larger |
| Needle 2 (45M) | 63.7% | 70 | baseline |
| Apple FM | 57.6% | ~6,000 | much larger |
Read that table honestly and Needle 2 comes third of four on accuracy. It loses to LFM2.5 230M by more than five points and edges just below FunctionGemma. It beats Apple's foundation model while using roughly one eightieth of the compute per token.
So the claim to take away is not "as good as models 70 times its size." It is very close to models five to six times its size, at a fraction of the compute, in a binary small enough to embed anywhere. That is still a genuinely impressive result. It is just a different claim, and the difference matters when you are deciding what to build on.
What testers found on day one
The model was posted to Hacker News on 10 August 2026, where it reached 535 points, and people started breaking it immediately. The thread is more useful than any benchmark for understanding the real edges.
The reported failures follow a clear pattern. Asked to "make it a little warmer in here," the model set the thermostat to cool mode at 65°F. Relative adjustments generally struggled, with "5 degrees warmer" being taken literally as 65 rather than applied to the current temperature. Given the input "HN," it attempted to lock the front door. Given "I'm hungover," it also tried to lock the front door, this time with zero confidence.
The Cactus team engaged directly and did not dispute the reports. They acknowledged the failures, pointed to their fine-tuning and data synthesis pipeline as the intended path to production quality, and noted the model performs better when tool descriptions are written clearly. Several commenters raised the obvious challenge, which is that if a model cannot reliably distinguish "warmer" from "cooler," a regular expression might do the job with fewer moving parts.
Do not ship the base model against open-ended user speech and expect it to hold up. Every failure above came from an unconstrained instruction with a broad tool list. The base model is a starting point that Cactus explicitly expects you to fine-tune on your own tools and phrasing, not a finished product you drop into an app.
The pattern that makes it useful in production
Local first, escalate when unsure
Buried in the Hacker News replies is the design idea that makes the whole thing work, and it reframes those failures entirely.
Needle 2 returns a confidence score with every tool call. The Cactus team's explanation is that this score is meant as a threshold signal: below your threshold, you do not act locally. You escalate the request to a cloud model or a larger local one.
That turns the model from "an assistant that is sometimes wrong" into "a cheap first pass that knows when to hand over." The hungover example is instructive here, because the model returned zero confidence on that call. A system wired correctly would never have executed it.
This is the same architectural instinct behind the three-agent split in Google's AMIE video consultation system, which we broke down in our piece on real-time AI agent architecture. A small fast model handles the immediate loop, something slower and stronger handles the hard thinking, and the design question is what the fast layer is allowed to decide on its own.
Where this fits with your existing stack
In a real product, Needle 2 sits in front of your cloud model rather than replacing it. The routing looks roughly like this.
- The instruction arrives on the device. Needle 2 attempts a tool call locally, in milliseconds, with no network.
- Check the confidence score against your threshold. Set this conservatively at first and tune it against real logs.
- Above threshold, execute locally. No API call, no latency, no data leaving the device.
- Below threshold, escalate. Send it to your cloud model, which has the context and the reasoning capacity to handle the ambiguous case.
- Log both paths. The escalations are your fine-tuning dataset for the next iteration.
Start with the threshold set high enough that almost everything escalates, then lower it as your logs show where the local model is reliable. Shipping with an aggressive threshold and discovering the failure cases through user complaints is the expensive version of the same learning.
What this changes commercially
The cost model
A cloud tool-calling layer bills per request forever. An on-device one costs you engineering time once and then runs free, on hardware your customer already paid for. For a feature invoked constantly, such as a voice control layer or a routing step that fires on every user action, that difference compounds in a way that changes what is worth building.
Be careful not to overstate it though. You are not removing your cloud model, you are reducing how often it gets called. If your escalation rate is high because the local model is not well fitted to your tools, you have added engineering cost without removing much API cost. The saving is real but it is proportional to how much traffic actually stays local.
Privacy, latency and offline
These three matter more than the cost for most of the products we get asked about.
Privacy is the strongest argument. If the instruction never leaves the device, there is no third-party processor in your data flow for that step, which materially simplifies the conversation in healthcare, legal, financial and any consumer product handling something sensitive. Latency disappears, because there is no network round trip at all. Offline operation becomes possible, which matters for field tools, vehicles, wearables and anywhere connectivity is unreliable.
| Consideration | On-device (Needle 2) | Cloud model |
|---|---|---|
| Per-request cost | None after the build | Ongoing, scales with usage |
| Latency | No network round trip | Round trip plus inference |
| Works offline | Yes | No |
| Data leaves the device | No | Yes |
| Handles ambiguity | Poorly, by design | Well |
| Improves without your work | No, you fine-tune it | Yes, provider upgrades it |
| Best role | First pass and confident cases | Escalation and hard cases |
When to use it, and when not to
Good fits
Needle 2 earns its place when the instruction space is narrow and predictable and you control the tool definitions. Smart home and appliance control with a fixed device list. In-app voice commands over a known set of actions. Structured extraction from documents or messages where the schema is fixed. Wearables and hardware where there is no room for anything larger and no reliable connection. Anywhere a privacy requirement makes sending the text off-device a problem.
Structured extraction is worth calling out separately, because several Hacker News testers who were unimpressed by the general tool calling found the extraction genuinely promising. Pulling fixed fields out of text is a much more constrained task than interpreting open-ended speech, and it plays to the model's strengths.
Bad fits
Do not reach for it when the user's phrasing is open-ended and unpredictable, when the request requires multi-step reasoning, or when a wrong action is expensive or hard to undo. A model that occasionally locks your front door because you mentioned a hangover has no business controlling anything with real consequences without a confirmation step.
Also be honest about whether you need a model at all. If your commands are genuinely fixed, a parser will be more reliable, easier to test and simpler to debug. The model earns its complexity when phrasing varies, not when it does not. That same question, whether the intelligent version is worth it, is the one we work through in AI agent development: what to automate versus keep human, and the cheaper options are covered in AI features you can add without breaking the budget. The full AI Applications cluster goes deeper on each pattern, and our SaaS development work covers building these into a product properly.
Frequently asked questions
Can Needle 2 replace ChatGPT in my app?
No, and it is not trying to. It is a 45M-parameter model built to select a tool and fill in its arguments, not to write, converse or reason through problems. In a well-designed system it sits in front of a cloud model, handling the confident cases locally and escalating everything else.
Is it actually accurate enough to ship?
The base model is not, against open-ended speech. Public testers broke it within hours on ordinary phrases like "make it a little warmer in here," and Cactus's own answer is that production use requires fine-tuning on your own tools and phrasing. With a narrow tool list, clear tool descriptions and a confidence threshold that escalates uncertain calls, it becomes usable.
What hardware does it need?
Very little, which is the point. It needs no GPU or NPU. Cactus reports it running on Raspberry Pi 5 at 500 tokens per second decode, on ESP32-S3 microcontrollers, on VR headsets, and on sub-$200 Android phones at 300 to 700 tokens per second. The binary is 14MB and a session uses roughly 28MB of RAM.
Can I use it in a commercial product?
Yes. Needle 2 is Apache 2.0 licensed, which permits commercial use and modification without revenue sharing, and lets you ship it inside a closed-source product. Verify the licence terms in the repository yourself before committing, since licensing on open-weight models has been shifting.
How much money would this actually save?
It depends entirely on your escalation rate. Every request handled locally is an API call you do not pay for, so a feature that fires constantly and stays local most of the time saves meaningfully. If most requests escalate because the model is not fitted to your tools, you have added engineering cost without removing much spend. Measure the escalation rate before assuming the saving.
What is the confidence score for?
It is a threshold signal, not a quality guarantee. Cactus's stated intent is that when confidence falls below your chosen threshold, the system should escalate to a larger model rather than act. Several of the funniest reported failures returned very low confidence, meaning a correctly wired system would have caught them before anything happened.
Thinking about putting an AI feature on-device, or working out whether the local model is worth the engineering against a simple cloud call? Get a free scope and quote and we will give you an honest read, including when the simpler option wins.
Sources
Have a project like this in mind?
Tell us what you're building and we'll map out the scope, timeline and a fixed starting quote — no obligation.
Start your project
