The Accelerator-Optional Tier: When On-Premises AI Runs on CPUs and NPUs Instead of a GPU Cluster

On 10 August 2026, Nanox.AI announced that it had optimised its medical imaging AI application framework for Intel Core Ultra processors using the OpenVINO toolkit. The stated purpose was to support AI applications running on an on-premise edge device inside healthcare facilities, letting organisations evaluate and deploy CT imaging AI while keeping imaging data within their own infrastructure. Intel’s Core Ultra line provides what the companies describe as a heterogeneous architecture — CPU, GPU, and NPU compute on the same processor — and the stated benefits are local inference, reduced dependence on cloud connectivity, and deployment models that fit hospital infrastructure.

Read as a healthcare story, it is a point release. Read as an infrastructure signal, it is more interesting: a regulated organisation choosing an inference target that is not a datacentre GPU, for reasons that are as much about data locality and operational fit as about throughput.

Most enterprise AI plans jump straight from “we need local inference” to “we need a GPU cluster.” There is a tier in between, and skipping it distorts both the budget and the schedule.

Three tiers, not one

It is more useful to think about inference placement as three distinct tiers with different economics, rather than one capacity question.

The distortion happens because sizing is usually done for the eventual ceiling, then applied to the first workload. A document classification service that processes a few thousand pages a day does not need the same hardware as an interactive assistant serving two thousand concurrent users, and pretending otherwise is how a pilot ends up waiting on a procurement cycle it never required.

What actually runs without a datacentre GPU

The workloads that behave well on CPU or NPU inference share a profile: bounded input size, tolerant latency expectations, and modest concurrency.

• Embedding generation and reranking. Batch indexing is throughput-tolerant and often runs overnight. Reranking a shortlist of candidates is a small model over short inputs — see Embedding Models and Rerankers for Private RAG.
• Document classification and field extraction. Routing an invoice, identifying a document type, extracting structured fields from a form — high volume, small per-item cost.
• OCR and speech transcription. Long-established as accelerator-optional workloads, and frequently the first stage of an ingestion pipeline.
• Domain vision models. Purpose-built models for images, scans, or inspection frames are often far smaller than general-purpose ones. The Nanox.AI case sits here.
• Small language models for scoped tasks. Summarisation of a bounded document, structured rewriting, and constrained question answering over a narrow corpus, as discussed in Selecting a Local LLM for Enterprise Workloads.

What does not run in this tier is equally clear: large models, long-context reasoning over many retrieved documents, high concurrent interactive load, multi-step agent workflows that chain many generations, and any form of training or fine-tuning. Attempting them on CPU produces latency that users abandon, which is a worse outcome than not deploying.

Placement is a governance decision, not only a latency one

The reason to put inference inside a facility is often misdescribed as latency. In regulated environments the stronger arguments are usually different.

Data never leaves the building. For imaging data, patient records, control-system telemetry, or plant footage, the strongest control is that the working data has no network path out of the site. That is a structural property, not a policy statement.

Degraded-mode operation. A hospital, substation, vessel, or remote plant may lose connectivity and still need the workflow. A local inference target keeps the function available when the link does not. This is the same reasoning behind Air-Gapped AI Deployments in Restricted Networks.

Fit with existing infrastructure. Many facilities cannot host accelerated servers at all — no rack space, insufficient power, no suitable cooling, no on-site staff. An appliance-class device is the only deployment shape the site can actually accept.

The caveat matters as much as the benefit. Local inference is not automatically sovereign. If the index, the model artefacts, the management plane, or the telemetry stream leave the site, the data path is wider than the inference path. The distinction is worked through in Data Sovereignty vs Data Residency in AI Procurement.

The cost that moves from hardware to operations

A fleet of edge devices trades capital cost for operational complexity, and the trade is not always favourable.

One centralised cluster has a single model inventory, one patch cycle, one configuration baseline, and one evidence store. Two hundred sites have two hundred of each. The practical risks are version drift — sites running different model versions with no record of which produced a given output — configuration divergence, inconsistent access control, and audit records that never converge into something reconstructible.

Before committing to a distributed footprint, three questions are worth answering explicitly. How is a model version promoted to, and rolled back across, the fleet, and can you state which version produced any given output? Where do execution records live, and how are they collected from a site that was offline? And who is accountable for a device that a facility team physically controls but a central platform team is expected to govern? The model lifecycle mechanics are covered in Local LLM Model Governance, Versioning and Upgrades, and the evidence side in AI Agent Observability: Logs, Traces and Audit.

Answering these before deployment is the difference between a fleet and a collection of unmanaged appliances.

A practical sizing sequence

Rather than sizing for the ceiling, size for the workload in front of you and measure your way up.

1. Characterise the workload: items per hour at peak, input size distribution, acceptable latency at the ninety-fifth percentile, and concurrency.
2. Test the smallest model that passes your evaluation set, not the largest one available. Quality is measured against the task, not the leaderboard.
3. Measure on the hardware you already own before buying anything. Existing server CPU capacity establishes the honest baseline.
4. Escalate a tier only when a measured indicator fails — tail latency, queue depth, or a quality threshold the smaller model cannot meet.
5. Record the decision and the measurements. When someone asks in a year why a workload runs where it does, the answer should be evidence rather than habit.

How VDF AI approaches mixed inference targets

VDF AI treats the inference target as a routing decision rather than an architectural constant. VDF AI Router directs each request to an approved model and execution zone based on the workload and its policy class, so a classification step can run on a lightweight local target while a reasoning step goes to accelerated capacity — within the same governed workflow. VDF AI Networks keeps the execution trace intact across those targets, which is what makes a mixed-tier deployment auditable rather than merely distributed.

The Nanox.AI and Intel announcement is a useful reminder that the interesting question in on-premises AI is no longer only how much accelerated capacity to buy. It is which workload belongs on which tier, in which location, under which policy — and whether you can prove where each one ran.