Fun with Core AI, 9 years later

In October 2017 I wrote Fun with CoreML. I had to install Python 2.7 because coremltools didn’t support Python 3, and the big selling point was that you could drag a model into Xcode and it just worked.

This year Apple replaced Core ML with a new framework called Core AI. Google spent the same year renaming TensorFlow Lite to LiteRT, open-sourcing its GPU engine ML Drift, and shipping an agent framework for Android. The names changed on both sides, so I went through all of it to see what actually changed underneath.

I haven’t benchmarked these head to head yet. Every number below is the vendor’s own, and I link to where it comes from.

The map

The two platforms split the work differently. Here’s the stack, from your app down to the silicon.

Figure 1. The iOS stack on the left, the Android stack on the right. The dashed box is the part of Google's stack that also runs on iPhone.

On iOS there are 2 layers you care about:

Foundation Models gives you a language model session with tools, images and agents. It talks to Apple’s on-device model, or to any provider you plug in.
Core AI runs your own model, of any kind, on the CPU, GPU and Neural Engine.

On Android there are more moving parts:

ADK for Android builds agents that can mix cloud and on-device models.
Gemini Nano is Google’s on-device model. It lives in a system service called AICore, and you reach it through the ML Kit GenAI APIs.
LiteRT runs your own model on the CPU, GPU or NPU. LiteRT-LM is the layer on top of it for language models.
ML Drift is the GPU engine inside LiteRT. You can also use it on its own.

Apple: Core AI

Core AI is the inference framework behind Apple Intelligence, now open to the rest of us.

Apple’s own description in the Meet Core AI session is that it scales from small models like speaker diarization up to “70B parameter LLMs”, with no server and no cost per token. (The 70B end is a Mac story. Don’t try that on an iPhone 😃)

This is how a model gets from PyTorch onto a device:

Figure 2. The Core AI pipeline. The first 4 boxes run on your Mac. The last box runs on the user's device, once.

Compress the PyTorch model with coreai-opt, if it is too big.
Export the model with torch.export.
Convert the exported program with coreai_torch.
Save the result as an .aimodel file.
Compile the model ahead of time on your Mac.
Let the device specialise the model for its own hardware on first load.

The compression options are int4, int8, FP4 and FP8, plus 4-bit palettization (weights stored as a small lookup table), which Apple recommends for power efficiency on iOS.

In the authoring session Apple took Meta’s SAM3 segmentation model from over 3 GB down to 430 MB with int4. It then lost a flower it used to detect, so they compressed only the encoders (96% of the weights) and left the detector alone. Quality came back.

The debugger is the part I like most. It compares every stage of the converted model against the PyTorch original and colours the stages that drift.

Apple’s presenter called it turning “hours of manual tensor comparison into a visual diagnosis in minutes”. In 2017 I did that comparison with print statements.

The Swift side is small:

import CoreAI

let model = try await AIModel(contentsOf: modelURL)
let main = try model.loadFunction(named: "main")!

var outputs = try await main.run(inputs: ["features": inputNDArray])
let logits = outputs.remove("logits")?.ndArray

There’s more under the hood for language models. Core AI keeps state between calls, so a transformer can hold its key and value cache instead of re-reading the whole history on each token.

You can split one model into several functions: Apple’s SAM3 demo reused the image embedding and got the second answer 76% faster. And you can ship your own Metal kernel inside the .aimodel file.

Watch the first launch. Do not specialise a model while the user waits. Specialisation is a full compile for that device. Compile ahead of time, specialise in the background with AIModel.specialize(contentsOf:), and show a “preparing” state until the cache has the model.

Foundation Models, the other half

If you only need a language model, you might never touch Core AI. The Foundation Models framework gives you the on-device model behind Apple Intelligence through a Swift API. This year it learnt 3 new tricks:

Any language model can sit behind the same API, including Claude and Gemini, through a LanguageModel protocol. A Core AI model can plug in the same way.
Prompts can carry images, and the model can call Vision tools like OCR and barcode reading on the device.
An Evaluations framework tests AI features beyond what unit tests cover.

There’s also a nice deal for small developers. If you’re in the App Store Small Business Program and have fewer than 2 million first-time downloads, the next Apple model on Private Cloud Compute costs you nothing.

Core AI, the good and the bad

Good

One framework, one toolchain and one debugger, from conversion to Instruments.
It uses all 3 compute units (CPU, GPU, Neural Engine) without you picking one.
The compression tools and the debugger are first class, and they come from Apple.
Language model support (state, multi-function models) is built in, and it plugs into Foundation Models.

Bad

Apple platforms only. Your Android team needs a second pipeline.
It’s 4 months old. Expect sharp edges and thin docs (some things haven’t changed since 2017).
First-run specialisation costs time on the user’s device, and you have to design around it.
Apple didn’t publish speed figures that I could find, so you can’t compare it with Google’s numbers without running it yourself.

Core ML is still in the SDK and Apple hasn’t announced a deprecation date that I could find. If you have a working Core ML model in production, there’s no fire.

Google: the Android stack

Google ships 4 pieces where Apple ships 2. Each one makes sense on its own, but it took me a while to see where each one stops.

LiteRT

LiteRT is the new name for TensorFlow Lite, and the runtime for your own models on Android. Google says its GPU path is 1.4x faster than TFLite.

The big news this year is the NPU.

In April Google moved NPU support into production for Google Tensor, MediaTek and Qualcomm chips, behind one API. Before that, each vendor had its own SDK (ask any Android developer who tried it 😉). The early results from Google’s partners:

Argmax got over 2x speed going from GPU to NPU for speech, across all 3 chip families.
Google Meet runs a segmentation model 25x larger without losing speed.
Epic’s Live Link Face drives MetaHuman animation at up to 30 FPS.

You can compile for the NPU ahead of time for a known chip, or on the device. Ahead of time means faster start-up and less memory, but one build per chip family. Google Play’s AI Packs then deliver the right build to each phone.

ML Drift

ML Drift is the GPU engine inside LiteRT. Google open-sourced it under Apache 2.0 on 8 October, a week ago as I write this.

It runs on OpenCL, OpenGL ES, Metal and WebGPU. So the same engine runs on Android, iOS, the web and macOS, with Windows and Linux in preview. The paper says it runs generative models with 10 to 100 times more parameters than earlier on-device GPU engines could.

Figure 3. Prefill reads the whole prompt in one parallel pass. Decode then writes one token at a time and reads the cache on each step.

ML Drift treats the 2 stages of a language model differently, and plans each one on its own. Prefill is compute-heavy and parallel. Decode is memory-heavy and serial, so I think one plan for both would leave the GPU waiting on memory half the time.

Google’s numbers from its own apps and partners:

Where	Result
YouTube Shorts	up to 40% lower average frame latency
Google Photos	up to 2 s faster than the legacy GPU delegate
Adobe	up to 30% faster photo editing on the device
Snapchat	30% lower latency for face and style effects

The old TFLite GPU delegate gets no new features from here on. If you use it, plan the move.

LiteRT-LM

LiteRT-LM is the layer for language models, built on LiteRT. It’s Google’s runtime for Gemma 4 (the E2B and E4B sizes, with vision and audio) on Android, iOS, the web, Chrome and Pixel Watch.

Google’s decode speeds on the GPU are 52 tokens per second on a Samsung S26 Ultra and 56 on an iPhone 17 Pro. (Yes, Google’s iPhone number is the higher one.)

Multi-token prediction adds up to 2.2x more. And the 2.58 GB Gemma 4 E2B model runs in 607 MB of memory on an Apple mobile CPU.

Gemini Nano and ML Kit GenAI

Gemini Nano is the Android equivalent of Apple’s on-device model. It runs in AICore, a system service that downloads and updates the model for you, keeps no record of requests, and has no direct internet access.

You reach it through ML Kit’s GenAI APIs. There’s an open prompt API, and 5 ready-made tasks: summarisation, proofreading, rewriting, image description and speech recognition.

Google says Gemini Nano is on over 140 million devices. That’s a lot of phones, and a minority of Android phones, so every feature needs a fallback.

ADK for Android

ADK (Agent Development Kit) is Google’s open-source agent framework. Kotlin and Android versions arrived at Google I/O in May, at version 0.1.0.

You mark ordinary Kotlin functions with @Tool and @Param, and a KSP code generator writes the JSON schema the model needs. Then you build agents from models and tools:

val onDeviceModel = GenaiPrompt.create(
    generativeModel = generativeModel, // ML Kit GenAI, Gemini Nano
    name = "gemini-nano",
)

The interesting part is the hybrid setup. A cloud Gemini model can act as the orchestrator, and hand privacy-sensitive jobs to sub-agents that run Gemini Nano on the phone.

Figure 4. The cloud orchestrator plans the trip. The on-device sub-agent reads the user's calendar and passport details, and only its answer leaves the phone.

Keep API keys out of the app. Google’s own docs say so. Call cloud models through your own backend or Firebase AI Logic.

The Android stack, the good and the bad

Good

The NPU is now usable on 3 chip families through one API.
Most of it is open source, and ML Drift and LiteRT-LM run on iPhone too.
ADK is the only one of these with a proper agent framework, cloud and on-device in one graph.
Gemini Nano updates through AICore, so the model improves without an app update.

Bad

4 pieces with overlapping names. It takes a while to see where each one stops.
Hardware varies. Ahead-of-time NPU builds mean one build per chip family.
Gemini Nano covers 140 million devices, a minority of Android phones.
ADK is 0.1.0. I wouldn’t put it in front of paying users without a fallback.

How they stack up

	Apple	Google
Your own model	Core AI	LiteRT (plus LiteRT-LM for language models)
Built-in language model	Apple Foundation Model	Gemini Nano in AICore
Language model API	Foundation Models	ML Kit GenAI
Agents	Foundation Models (tools, swappable models)	ADK for Android 0.1.0
Cloud models in the same API	Yes, through `LanguageModel`	Yes, through ADK
Model file	`.aimodel`	`.tflite`
Conversion	PyTorch through `coreai-torch`	PyTorch, JAX or TensorFlow to LiteRT
Hardware	CPU, GPU, Neural Engine	CPU, GPU (ML Drift), NPU (Tensor, MediaTek, Qualcomm)
Runs on the other platform	No	Yes, LiteRT, ML Drift and LiteRT-LM run on iOS
Open source	The coreai-models repo	LiteRT, ML Drift (Apache 2.0) and ADK
Published speed figures	None that I found	Many, from Google and partners

I think 3 differences matter most.

Apple builds for one hardware family, Google builds for hundreds. That’s why Apple ships 1 framework and Google ships 4. Apple’s version is easier. Google’s version gives you more control, and you need that control because the phones are so different.

Google’s stack runs on iPhone. If you want one model and one runtime on both platforms, LiteRT-LM is the only option here. You give up the Neural Engine on iOS, because LiteRT-LM’s iPhone figures are for the GPU and CPU.

Both now treat the cloud model as a plug-in. Apple does it inside Foundation Models, and Google does it in ADK. The on-device model handles private and offline work, and the cloud model handles the hard reasoning. I expect most real apps to end up hybrid.

Use cases

Summarise private notes offline. A notes or email app that summarises without sending anything out. On iOS use Foundation Models. On Android use the ML Kit GenAI summarisation API, and fall back to a cloud call (with the user’s permission) on phones without Gemini Nano.

Photo and video effects with your own model. On iOS convert the model with Core AI, and compress it with palettization like Apple’s SAM3 demo. On Android run it on LiteRT with ML Drift. Adobe, Snapchat and YouTube Shorts are doing this now.

Live speech to text. Argmax got over 2x on the NPU with LiteRT. On iOS, Apple names speaker diarization as a Core AI model, and the Neural Engine is the natural home for an always-on audio model.

A travel agent that knows your passport. This is Google’s own ADK example: a cloud orchestrator plans the trip, and on-device sub-agents handle the personal details. On iOS you can build the same thing with Foundation Models by swapping between the on-device model and a cloud model within one session.

Real-time avatars and face tracking. Epic’s Live Link Face hits up to 30 FPS on the NPU. Anything that runs per camera frame belongs on the NPU or the Neural Engine, for the battery as much as for speed.

One language model on both platforms. Ship Gemma 4 with LiteRT-LM on Android and iOS, and keep one prompt and one set of evals. You pay for it with a 2.58 GB download, so deliver it after install (AI Packs on Android, background assets on iOS).

So which one?

Figure 5. My rule of thumb. Start at the top and stop at the first yes.

If the built-in model can do the job, use it: Foundation Models on iOS, ML Kit GenAI on Android.
If you need your own model on one platform, use that platform’s runtime: Core AI on iOS, LiteRT on Android.
If you need the same language model on both platforms, use LiteRT-LM.
If you need an agent that mixes cloud and on-device models, use Foundation Models on iOS and ADK on Android.

What I want to test next

The vendor numbers don’t answer the question I care about. How does Gemma 4 on Core AI, with the Neural Engine, compare with Gemma 4 on LiteRT-LM, on the same iPhone? I couldn’t find anyone who has published that yet, and it’s the test I want to run.

If you’ve shipped any of these in a real app, I’d love to hear what broke first.

Thanks for reading 😃

← All writing