geisten research

Back to research

AI Everywhere Requires Local AI

Perspective 25 July 2026 9 min read Dr. Germar Schlegel

AI everywhere is not a chatbot in every object. It means placing the smallest useful intelligence where the work happens. Our conclusion is local-first, not local-only: keep bounded tasks close to the device, escalate deliberately and measure the complete system.

Measured systems for practical intelligence - geisten

Imagine a workshop camera that notices a damaged part, a hearing aid that separates speech from noise or a home assistant that understands a private request. Where should the intelligence for these tasks run?

The usual answer is: somewhere else.

Data travels to a large model in a data centre. The result travels back. This works remarkably well when the network is available, the delay is acceptable and the data is allowed to leave the device. But those conditions are not laws of nature.

If artificial intelligence is to become part of ordinary objects and infrastructure, we need another option. Useful inference must also fit inside phones, homes, machines and inexpensive computers.

The question is not whether local models will replace cloud models. A more useful question is:

How do we decide where each intelligent operation should run?

What does “AI everywhere” actually mean?

It does not mean putting a general-purpose chatbot into every product.

A temperature sensor does not need to write poetry. It may only need to notice an unusual pattern. A document workflow may need extraction and classification, not open-ended conversation. A small robot may need fast local perception and only occasional help with planning.

The useful unit is therefore not “one large model”. It is a task with a quality threshold and a physical environment.

That changes the engineering goal. We are no longer asking for the highest possible benchmark score at any cost. We are asking for enough capability within limits on response time, memory, energy, data movement and money.

Our first conclusion follows from that change:

AI everywhere requires the smallest system that can perform a defined task reliably, not the largest model we can reach.

Sometimes that smallest system will be a classifier. Sometimes it will combine rules, retrieval and a compact language model. Sometimes only a frontier model will meet the quality requirement. The architecture should begin with the task, not with the prestige of the model.

Are small models becoming useful quickly enough?

There is good evidence that the design space is expanding.

According to Epoch AI, the leading open-weight models lagged the state-of-the-art closed-model frontier by an average of four months from January to May 2026 on the Epoch Capabilities Index. The average score gap was eight points. This is a composite benchmark result, not proof that an open model is equally good for every application. Epoch also notes that public benchmark optimisation and incomplete evaluation access may distort the comparison.

The important observation is narrower: developers increasingly have capable models whose weights they can inspect, adapt and run on infrastructure they control.

The cost of capability has also fallen. The 2025 Stanford AI Index reports that the API price of a model reaching GPT-3.5-level performance on MMLU fell from USD 20 per million tokens in November 2022 to USD 0.07 in October 2024. This number describes one benchmark threshold and historical API prices. It does not predict the cost of a particular application. It does show how quickly yesterday’s expensive capability can become ordinary.

On-device systems provide a more concrete signal. Apple describes an approximately three-billion-parameter language model designed for tasks such as summarisation, extraction and text understanding. Its weights are compressed to two bits per weight, while its cache and architecture are shaped around device constraints. Apple explicitly says that the model is not intended to be a general world-knowledge chatbot.

That limitation is the interesting part. The model has a job, and its design follows from that job.

We have seen the same principle on much older hardware. In a Geisten experiment, a 2017 OnePlus 5T ran a compact Qwen 2.5 model locally. It did not become a frontier AI system. The experiment showed that model format, runtime and task definition can make useful inference possible on hardware built years before the current on-device AI wave.

Our second conclusion is therefore not that small models have caught up with the largest systems. It is this:

The set of useful tasks that can run locally is growing, even while frontier models continue to improve.

That is enough to change system design. Local AI does not need to win every benchmark. It only needs to become the better component for a particular part of the workflow.

What do we gain by moving inference closer?

The strongest case for local AI is not a single feature. It is the combination of four properties.

Shorter paths. A local system can avoid a network round trip. This matters when a machine must react quickly or when an interaction contains many small steps. The relevant metric is end-to-end latency, not a theoretical hardware peak or an isolated tokens-per-second number.

Offline operation. A device can continue to perform its bounded task when the network is weak, unavailable or intentionally disconnected.

A smaller data boundary. Audio, images, documents or sensor readings may remain on infrastructure controlled by the user. This can reduce exposure, but only if telemetry, logs, retrieval, tools and update services respect the same boundary.

Operational control. The operator can decide when to update a model, which version to run and what happens if a provider changes a service. Open weights can increase this control, but model origin, licence and deployment location are separate questions.

None of these benefits arrives automatically. A local model connected to cloud tools may still disclose sensitive information. A poorly cooled device may throttle. A slow local answer may be less useful than a fast remote one. And software updates can create a new dependency even when inference itself is offline.

Our third conclusion is:

“Runs locally” is not a result. It is an architectural property whose value must be demonstrated at the boundary of the complete system.

Is local AI cheaper or more energy-efficient?

There is no honest universal answer.

A cloud API turns infrastructure into a variable cost. It can be economical for rare requests, difficult tasks or rapidly changing requirements. A shared accelerator may also serve many users efficiently.

A local device has different economics. Hardware may already exist. Repeated inference can avoid API and network costs. But purchase price, utilisation, electricity, cooling, maintenance, integration and security updates still count.

Energy comparisons are equally easy to misstate. A smaller model file does not prove lower energy use. Neither does a lower instantaneous power reading. A slow device can draw little power and still consume more energy for one result because it runs longer. A fair comparison needs the energy for the complete task and a documented measurement boundary. For a cloud system, that boundary must also account for the parts of the remote service that are actually known; often they are not.

This makes the fourth conclusion deliberately conditional:

Local AI is cheaper or more efficient only for a defined workload, at a defined scale and within a stated system boundary.

The conclusion may change when request frequency, hardware utilisation, quality requirements or network conditions change. That is not a weakness of the comparison. It is the reason to measure instead of generalise.

When should the cloud still win?

Some tasks require knowledge, context or reasoning that a local model cannot provide reliably. Others happen too rarely to justify dedicated hardware. Cloud systems can offer stronger models, elastic capacity and rapid updates.

The mistake is not using the cloud. The mistake is using it without making the decision visible.

A local-first architecture starts close to the task and escalates when there is a reason. The route may look like this:

  1. Use deterministic software for deterministic work.
  2. Use a small local model for a bounded task with a known quality threshold.
  3. Ask for permission, remove unnecessary data and escalate when the local result is insufficient.
  4. Record why the escalation happened and whether it improved the outcome.

This is not local-only. It is selective escalation.

The difference matters. In a cloud-first system, every request leaves the device unless an exception is built. In a local-first system, data remains close to its source unless the task justifies a wider boundary.

Our position is therefore:

Deployment location should be a first-class design decision, just like model quality, latency and cost.

What evidence would change our mind?

Local AI becomes an engineering discipline only when its claims can fail.

For every target device and task, we should ask:

The answers may show that a compact model is too inaccurate, that quantisation hurts performance on an older processor or that a shared service uses less energy for an infrequent task. Those are useful results. They tell us where the local boundary currently lies.

Conclusion

AI everywhere will not be built by copying the same remote chatbot into every product.

It will be built by decomposing useful behaviour into tasks, placing each task where it can run reliably and moving data only when the benefit is clear. Smaller models, open weights and efficient runtimes make more of those tasks local candidates. They do not make the cloud obsolete.

Our conclusion is local-first, not local-only:

The cloud should remain an option. It should no longer be the automatic answer to where intelligence lives.

Limitations

This is a perspective article, not a new benchmark or a forecast. The examples show that smaller and locally deployable models are becoming useful for more tasks; they do not establish quality, cost or energy advantages for an arbitrary application.

The cited model rankings and prices are time-sensitive. Composite benchmarks do not predict quality for a specific workflow. Vendor documentation describes vendor systems and is not an independent comparison. The Geisten smartphone experiment demonstrates technical feasibility on one device and task, not a general performance claim.

Sources