KorgisActive developmentINFRASTRUCTURE · LOCAL AIActive development · v0.8.xOne stable local inference boundary for applications, while models, runtimes, routing and lifecycle can evolve underneath.
WHY IT EXISTS
The infrastructure problem starts after the weights load: applications still need a stable contract, lifecycle ownership, concurrency rules and operational visibility.
Model identity, configuration and dynamic routing are explicit and reproducible across environments.
Server health, resident state, token generation timings and logs remain fully inspectable.
Multiple desktop applications can depend on the exact same OpenAI-compatible local reasoning layer.
Executing model weights is straightforward. Turning open-weight models into a stable execution layer for production software requires solving core system engineering problems.
Application logic directly inherits inference engine quirks, quantization formats, and binary execution flags.
Models must be resolved, completeness-checked, loaded into VRAM, drained of active requests, and unloaded safely without memory leaks.
Running multiple resident models simultaneously requires explicit routing, admission control, and hardware resource concurrency limits.
Product teams need request evaluation timings, streaming health indicators, and log tracing beyond raw stdout console output.
Applications integrate once using the standardized OpenAI protocol. Models, backend engines, and quantization choices evolve independently behind the runtime boundary.
Product code issues a standard /v1/chat/completions HTTP request.
Request parameters, prompt templates, and streaming flags are parsed and validated.
The server resolves the model key to a configured GGUF or MLX definition.
Acquires a concurrency lease on the active model, preventing unsafe unloads.
Executes inference via in-process llama_cpp/mlx or managed server processes.
Delivers structured JSON output or low-latency Server-Sent Events stream to the client.
FastAPI boundary, ModelRuntimeManager state coordination, and pluggable backend engine adapters.
Handles HTTP CORS, OpenAI API route matching, request validation, and SSE response streaming.
Central orchestrator managing built-in and user model registry merges, lease admission, and state.
Managed child backend server processes are monitored with SIGTERM/SIGKILL bounded timeouts.
Tracks prompt evaluation milliseconds, generation speed (tok/s), and active memory footprint.
Downloaded does not mean resident, and resident does not mean default route. Lifecycle state changes are managed deterministically.
Discovers models from built-in definitions, user configs, LM Studio caches, or Hugging Face hubs.
Validates file existence, SHA-256 integrity checksums, and quantization compatibility before loading.
Allocates Metal/VRAM buffers, context windows, and threads for the target backend.
Maintains active client leases to guard against premature model unloads during active inference.
Drains inflight HTTP requests before releasing memory back to the host system.
Switch between optimized execution backends based on hardware architecture and model capabilities.
| Backend Engine | Format | Execution Model | Current Role & Target Workloads |
|---|---|---|---|
llama_cpp | GGUF | In-process (C++ bindings) | Text generation, structured reasoning & quantized GGUF models |
mlx | MLX | In-process (Apple Metal) | Ultra-fast Apple Silicon text inference with Metal memory unified architecture |
llama_server | GGUF + Projector | Managed process | Multimodal and audio-capable GGUF models requiring separate server process |
mlx_vlm_server | MLX VLM | Managed process | Apple Silicon vision-language model inference via python MLX-VLM package |
Point any OpenAI-compatible client library to the local server port.
from openai import OpenAI
# Connect to Local LLM Server running on localhost
client = OpenAI(
base_url="http://127.0.0.1:1235/v1",
api_key="local", # Any non-empty string
)
# Execute streaming chat completion
response = client.chat.completions.create(
model="nemotron-nano-4b-q8",
messages=[
{"role": "user", "content": "Extract decisions and action items in JSON."}
],
stream=True,
)
for chunk in response:
content = chunk.choices[0].delta.content or ""
print(content, end="", flush=True)Korgis is the bundled operational web console that makes the runtime server visible, testable, and easier to integrate.
Test prompts, stream responses, configure inference parameters, and inspect structured outputs.
Inspect resident model leases, discover local GGUF/MLX registries, and adjust backend context sizes.
Follow prompt evaluation latency, generation tok/s throughput, and detailed runtime process logs.
Move from inspection to application code with copy-ready OpenAI client SDK code and live Swagger specs.
EVIDENCE
A strong infrastructure claim includes the boundary of the claim. Local LLM Server documents what is integrated today without turning active development into an implied production guarantee.
EVIDENCE BOUNDARY
The local runtime boundary is implemented; open-network production readiness is not claimed.
The current server supports local model discovery, multi-model routing, OpenAI-compatible streaming, explicit lifecycle handling and an inspectable Studio surface. Its default trust boundary remains loopback-first, and untrusted remote deployment requires controls the project does not currently provide.
Supported today
Not claimed here
STATUS & NEXT DECISION
The next step is to deepen representative hardware validation and operational hardening without pretending that a loopback-first developer runtime is already trusted remote infrastructure.
CURRENT DECISION
Use the OpenAI-compatible boundary locally; do not imply trusted remote infrastructure.
Local LLM Server binds to 127.0.0.1 by default and already supports multi-model routing, streaming, lifecycle management and operational visibility. It does not include a built-in authentication layer for open or untrusted network deployments, and hardware-specific tuning remains part of responsible use.
CONNECTED SYSTEM
Explore the runtime codebase, benchmark against your desktop hardware, and adopt an OpenAI-compatible boundary for your local AI applications.