1
Duplicated Engine Setup
Every app bundles its own runtime binaries, inflating APK size and multiplying maintenance overhead.
Android LLM HarnessSpec & Prototype
GitHub ↗From AI strategy to reusable infrastructure
Build local AI once.
Reuse it across Android.
Harness is a Local AI Gateway for Android: one foundation for GGUF models, on-device inference, runtime lifecycle and diagnostics.
Local-first executionExplicit model identityMeasured performance
Android API 26+GGUFllama.cppKotlin + JNI/C++MIT
Core Gateway Pattern: App → IPC → Gateway → EngineGGUF / llama.cpp
Project Mission
Make local AI on Android explicit, measurable and reusable.
Running LLMs on-device shouldn't mean re-implementing model loading, memory limits, and runtime bindings in every single app. Harness abstracts on-device inference into an explicit system service.
ExplicitNo opaque black boxes
MeasurableTokens/sec & RAM tracking
ReusableOne gateway for N apps
The Challenge
Why on-device AI needs a Gateway layer
Building local LLM features on mobile today comes with structural friction that impedes product reliability.
2
Uncontrolled Memory Spikes
On-device LLMs consume gigabytes of RAM. Without centralized management, apps crash silently under OOM conditions.
3
No Standard IPC Protocol
Integrating local AI into existing background services or third-party apps requires custom ad-hoc protocols.
4
Opaque Performance Metrics
Measuring prompt evaluation speed, generation latency, and thermal throttling requires dedicated telemetry.
Concept
Think of Harness as the local AI layer of an Android device.
Just as Android provides system services for location, camera, or network, Harness provides a system gateway for local LLM execution.
01
Unified Model Repository
Store and manage GGUF quantized models in a centralized device location shared across applications.
02
High-Performance C++ Runtime
Direct bindings to llama.cpp via JNI with hardware acceleration for ARM NEON and Vulkan GPU execution.
03
Standardized AIDL & IPC
Expose thread-safe Android Binder interfaces so any app can stream tokens without runtime lock-in.
Connected console
Five product surfaces. One inference foundation.
Harness connects runtime controls, model management, interactive testing, diagnostics and system settings into a unified mobile console.
Overview
Runtime readiness, device resources and next valid action.
Playground
Prompt, configure, stream and cancel inference locally.
Models
Import, download, verify, select and remove GGUF artifacts.
Diagnostics
Runs, health, resources, logs and benchmarks.
Settings
Privacy, storage, build information and developer controls.
Security Boundary: Current boundary: the runtime is still embedded and in-process inside Harness. Other Android apps cannot connect to a shared Binder/AIDL service yet. The gateway is the target architecture, not a current capability.
Under the hood
Start simple. Then go technical.
The architecture is intentionally layered so the product surface, runtime contracts, model plane, native backend and observability can evolve without leaking implementation details into every client.
Android appproduct / client
LocalLlmClientpublic boundary
RuntimeOrchestratorlifecycle + scheduling
Model resolutionapp + use case
Model storeverified GGUF
InferenceBackendreplaceable backend
llama.cppJNI / C++17
Full technical architecture
Design values
Infrastructure designed for decisions, not demos.
The product is organized around the qualities that make local AI useful after the first successful generation.
Local-first control
Inference runs on-device and normal telemetry excludes prompts and generated output.
Reuse instead of duplication
The gateway vision centralizes model storage and resource ownership so multiple authorized apps can reuse infrastructure.
Evaluation before confidence
Latency, throughput, memory, thermal state, health and benchmarks make model choices inspectable.
Guardrails that enable delivery
Bindings, integrity, cancellation, failures and cleanup are explicit rather than hidden runtime assumptions.
Open, replaceable boundaries
Product contracts stay independent from llama.cpp internals, Android UI and transport details.
Shared operational visibility
The target gateway adds one privacy-conscious control plane across local AI applications.
Strategy
From console to gateway.
The sequence deliberately proves difficult runtime and model-management foundations before introducing cross-application IPC and shared ownership.
1 · Local AI Console
Prove the data plane
Distribution, storage, inference, cancellation, cleanup, diagnostics and real-device behavior.
2 · Stable client boundary
Stabilize the contract
Keep explicit application/use-case identity and backend-neutral domain contracts.
3 · Local AI Gateway
Move ownership
Protected Binder/AIDL transport plus a lightweight Android client integration.
4 · Multi-app platform
Unlock shared value
Shared models, RAM coordination, scheduling, health and monitoring across authorized apps.
Developer boundary
Client code stays small on purpose.
Applications depend on a neutral client and explicit identifiers rather than calling JNI directly. The target gateway keeps this interaction while moving backend ownership behind the host service.
val prepared = client.prepare(applicationId, useCaseId)
check(prepared.ready)
val sessionId = client.createSession(applicationId, useCaseId)
val handle = client.generate(
request = generationRequest(sessionId, applicationId, useCaseId),
listener = generationListener,
)
handle.cancel()Evidence & maturity
Useful today. Still an engineering and validation project.
Harness should not be presented as a production-ready Android inference platform yet. The current value is a connected console and tested foundation that de-risks the final gateway architecture.
Integrated today
- Embedded GGUF inference path
- Streaming and cancellation
- Model import, verification and lifecycle
- Connected console surfaces
- Telemetry, health, resources and benchmarks
Still blocking production claims
- Representative physical-device GGUF evidence
- Memory and thermal validation on target devices
- Packaging and production-readiness evidence
- Cross-application Binder/AIDL host is still future work
Open-source · MIT
Make local AI explicit, measurable and reusable on Android.
Harness is part of a broader mission: move beyond isolated AI experiments and build product infrastructure that can be understood, evaluated and scaled deliberately.