One control plane for every engine

Orchestrate LLM inference
across your fleet.

Open-source inference orchestration, written in Rust. It sits in front of the engines you already run and sends each request to the node that already has its prefix cached. The same setup runs on a few DGX Sparks, AMD boxes, or Mac Studios on your desk and on a Kubernetes or cloud fleet. Install is one curl command and six binaries.

Get Started GitHub Docs

RustOpenAI APIvLLMSGLangllama.cppMLXcgn-inferTRT-LLM driverBare metalKubernetesQUIC

Overview

The inference orchestration layer

Cognitora runs on top of your inference engines and joins them into one distributed cluster. You can use it in place of an orchestrator like NVIDIA Dynamo. Compared with Dynamo, it supports more engines (llama.cpp, Apple MLX, any OpenAI-compatible server), deploys on bare metal without Kubernetes, factors power use into scheduling, and has no Python control plane.

Sub-millisecond routing

Each routing decision takes under 500µs, using sequence-chained BLAKE3 digests and longest-prefix overlap.

Engine-agnostic

One OpenAI-compatible gateway in front of vLLM, SGLang, llama.cpp, Apple MLX (mlx-lm), a TRT-LLM spawn driver, and the first-party cgn-infer engine.

Security

mTLS on every internal connection, Cosign-verified releases, and built-in OIDC, API-key, and RBAC auth.

Bare metal to fleet

Start on one node with systemd and six static binaries. Add QUIC federation when you outgrow a single rack.

Platform

Deploy only what you need

Each component deploys on its own. On bare metal that means six static binaries under systemd; add the operator if you run on Kubernetes.

cgn-router

KV-aware request router

OpenAI-compatible gateway. It scores nodes by longest-prefix overlap of BLAKE3 sequence digests, using the cache claims agents send in heartbeats; querying cgn-kvcached live is on the roadmap. Splits prefill from decode.

Learn more

cgn-agent

Engine supervisor

Runs on each node and supervises any engine with an OpenAI HTTP API. Ships drivers for vLLM, SGLang, llama.cpp, MLX, the native cgn-infer engine, and others, reports NVML telemetry, and hands off KV cache through NixlConnector. One cluster can mix engines.

Learn more

cgn-kvcached

Multi-tier KV cache

RAM/SSD tiered cache behind engine-owned GPU HBM, with cross-node QUIC fetch. Pluggable offload: nixl, lmcache (in-process or LMCache MP server), hicache, or kvbm.

Learn more

cgn-metrics

Power-aware telemetry

Aggregates Prometheus metrics along with Redfish and NVML power readings. The router sends less traffic to nodes that are running hot.

Learn more

cgn-ctl

Admin CLI

Bring up clusters, deploy models, shape traffic, and debug on bare metal. Helm charts are available if you deploy on Kubernetes.

Learn more

cgn-operator

Kubernetes operator (optional)

kube-rs operator with CRDs for InferenceCluster, ModelPool, and RoutingPolicy, for declarative rollouts on Kubernetes. Bare-metal installs don't need it.

Learn more

cgn-inferpreview

Native inference engine

First-party GGUF engine built on Candle (CPU, Metal, CUDA). It runs models in-process behind the same OpenAI HTTP API, so you don't need a separate engine server. Supports continuous batching and pipelining layers across nodes.

Learn more

Features

What it does

KV-aware routing

Requests go to the node whose cache holds the longest matching prefix, so the same prompt isn't prefilled twice. A node's cache claim only counts after a request completes there. Live overlap queries against cgn-kvcached are on the roadmap.

Tool calling and structured output

tools, tool_choice, and response_format pass through to the engine, including streamed tool-call deltas.

Multimodal input

OpenAI image content parts go straight to vision-capable engines. Prefix hashing only looks at text.

Prefill/decode split

Prefill and decode run on separate replica pools with KV handoff through NixlConnector, so each phase can use different hardware.

Multi-tier KV cache

RAM (DashMap), SSD (RocksDB), and cross-node QUIC peer fetch.

Energy-aware scheduling

Power readings from Redfish and NVML (rocm-smi on AMD) feed the routing score, with soft per-node watt caps. A closed-loop autoscaler drains hot nodes, and an SLA planner scales decode replicas against queue-depth SLOs.

Cross-cluster federation

Router forwards to the lowest-latency reachable peer cluster when no local node serves a model. KV cache peers via QUIC.

Gossip discovery without etcd

With state_backend = gossip, nodes form a UDP scuttlebutt mesh with phi-accrual failure detection, so a multi-node cluster needs no external services.

Multi-model cascade

A small model answers first, and the request moves to a larger model when confidence is low. Works on buffered and streaming requests.

Pluggable KV offload

One TOML knob: none, nixl, lmcache (in-process or a node-local MP server), hicache, or kvbm.

Engine-agnostic agent

Spawns and supervises vLLM, SGLang, llama.cpp, MLX, TRT-LLM (trtllm-serve), or any OpenAI-compatible server, and reports live queue and KV telemetry to the router.

Apple Silicon (MLX)

engine.kind = mlx spawns mlx-lm’s OpenAI-compatible server. Example: examples/apple-mlx/.

Cluster dashboard

A single HTML file that reads the Prometheus endpoints and shows throughput, latency percentiles, KV utilization, power, and J/token per node.

Native engine (cgn-infer)

First-party GGUF engine on Candle that runs models without an external server. Set engine.kind = cgn_infer. In preview.

Observability

Cluster dashboard

The repo includes a dashboard that is a single static HTML file. It reads the Prometheus /metrics endpoints directly and shows requests/s, tokens/s, latency and TTFT percentiles, queue depth, per-node status, KV-cache utilization, fleet power, and energy per token. You don't need Grafana or a backend to run it.

A 16-node mixed fleet (H100, H200, A100, MI300X, L40S, A10) serving five models under load.

Compatibility

Engine support matrix

Cognitora works with any engine that exposes the OpenAI HTTP API, but the features you get depend on the engine. TRT-LLM support is a spawn driver for trtllm-serve; it has no default recipe yet and no end-to-end GPU test in CI.

Feature	llama.cpp	MLX	cgn-inferpreview	OpenAI
OpenAI HTTP gateway
KV-aware routing
Prefill/decode split	—	—	—	—
KV offload: LMCache
KV offload: LMCache MP
KV offload: HiCache
KV offload: KVBM
Multi-tier KV (RAM/SSD)
Multi-model cascade
Energy-aware admission

configurable, waiting on upstream support. LMCache MP is the standalone lmcache server plus LMCacheMPConnector (vLLM). SGLang uses --enable-lmcache. · preview = cgn-infer, the experimental first-party native engine (Llama-family GGUF)

Architecture

Separate prefill and decode pools

Prefill and decode run on separate pools, and the router sends each request to the node that holds its cache. Components talk to each other over gRPC with mTLS.

Bare Metal

systemd on Linux

Kubernetes

Helm + operator CRDs

Cloud

Terraform: AWS, GCP, Azure, Hetzner

Local Dev

Docker, Ollama, MLX on macOS

Use Cases

Where it fits

Apple Silicon and edge

Run the same router stack on a Mac. The agent starts mlx-lm's OpenAI-compatible server on Apple Silicon, and KV-aware routing works as it does on a GPU cluster.

MLXmacOSDev laptop

Reasoning models

Put prefill and decode for MoE and chain-of-thought models on separate pools, and size each pool for its phase.

MoECoTLong context

Bare-metal GPU racks

Install with one curl command and run Llama on a DGX Spark or a mixed AMD/NVIDIA rack under systemd, with no Kubernetes.

systemdBare metalMixed GPUs

AI agents

Agents send the same context over and over, and KV-aware routing sends each follow-up to the node that has that prefix cached.

KV cachePrefix reuseLow TTFT

Code generation

Large repository contexts are cached in RAM, on SSD, and on peer nodes. Contexts in frequent use stay in memory, and the rest are fetched when needed.

Context reuseMulti-tierStreaming

Cost-optimized serving

A small model handles easy queries and hands off to a large one when its log-probability is low, so simple requests cost less.

SLM→LLMCostQuality gates

vs NVIDIA Dynamo

Where each one is stronger

Both orchestrate engines in a similar way; they differ in runtime and in how you deploy them. Cognitora is stronger on mixed hardware, energy-aware routing, and low-ops setups. Dynamo is stronger at NVL72 scale.

Bare-metal first

Install with one curl command and run six static binaries under systemd. Kubernetes and Terraform are optional.

Engine breadth

vLLM, SGLang, llama.cpp, MLX, any OpenAI-compatible server, and a TRT-LLM spawn driver, mixed in one cluster.

Energy-aware routing

Redfish and NVML power readings are part of the routing score, with soft watt caps and drain hints for the autoscaler.

Heterogeneous fleets

Per-model GPU constraints keep a 70B model off the L40S, so you don't need a separate deployment for each hardware class.

Where Dynamo leads today

Disaggregated multimodal & video pipelines
Native KVBM tiered block manager (G1 to G4)
ModelExpress GPU-to-GPU weight streaming
Grove NVL72 topology-aware gang scheduling
In-flight request migration
AIConfigurator deployment search

Still missing, planned for v0.5

Router ↔ cgn-kvcached overlap

local heartbeat scoring today

In-flight migration

pre-token retry only

Multi-node GPU E2E

outside default CI

RDMA transport

stub behind feature gate

Full comparison doc

Performance

CI-gated targets

CI checks every merge to main against these targets. The benchmark harnesses are in the repo.

Routing decision

< 500µs

per vCPU, CI-gated

HTTP overhead

< 3ms

p99 vs direct engine

Warm cache hit

< 200µs

RAM tier lookup

Cold cache hit

< 5ms

SSD/RocksDB tier

Cross-node fetch

< 12ms

1 MiB, QUIC, 10 GbE

Cache hit ratio

≥ 55%

representative traces

Energy efficiency

1.4×

vs round-robin

Model bring-up

< 30s

using recipes

Recipes

Production deployment profiles

Each recipe lives at model/engine/topology/ and has a 3-line up.sh that gets a model serving in under 30 seconds.

Model	Engine	Topology
Llama 3.2 3B (MLX)	MLX (mlx-lm)	Apple Silicon example
Llama-3.1 8B	vLLM	1×A100 aggregated
	vLLM	1×A100 disaggregated
	SGLang	1×A100
	vLLM + LMCache	1×A100 in-process
	vLLM + LMCache MP	1×A100 node-local server
	cgn-infer	Native GGUF single-node
Llama-3.3 70B FP8	vLLM	TP=4 multi-GPU
Qwen3-7B	vLLM	1×GPU
	SGLang	1×GPU
DeepSeek-v4-Flash	vLLM	Multi-GPU
	SGLang	Multi-GPU

$ bash recipes/llama3-8b/vllm/agg/up.sh

Quickstart

One command to install

$curl -fsSL https://inference.cognitora.dev/install | sh

Linux

# prebuilt binaries (x86_64 / aarch64; default = latest GitHub release)
# inference.cognitora.dev/install redirects to deploy/installer/install.sh on GitHub
curl -fsSL https://inference.cognitora.dev/install | sh

# pin version / prefix: vars apply to the shell running install.sh (see deploy/installer/install.sh)
curl -fsSL https://inference.cognitora.dev/install | CGN_VERSION=v0.9.2 sh
curl -fsSL https://inference.cognitora.dev/install | CGN_PREFIX=$HOME/.local sh

# verify
cgn-ctl version

Then send a request (Linux)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

Community

Try it on your hardware

Follow the quickstart to get a cluster running, or star the repo on GitHub.

Star on GitHub Quickstart

Orchestrate LLM inferenceacross your fleet.

The inference orchestration layer

Sub-millisecond routing

Engine-agnostic

Security

Bare metal to fleet

Deploy only what you need

KV-aware request router

Engine supervisor

Multi-tier KV cache

Power-aware telemetry

Admin CLI

Kubernetes operator (optional)

Native inference engine

What it does

KV-aware routing

Tool calling and structured output

Multimodal input

Prefill/decode split

Multi-tier KV cache

Energy-aware scheduling

Cross-cluster federation

Gossip discovery without etcd

Multi-model cascade

Pluggable KV offload

Engine-agnostic agent

Apple Silicon (MLX)

Cluster dashboard

Native engine (cgn-infer)

Cluster dashboard

Engine support matrix

Separate prefill and decode pools

Where it fits

Apple Silicon and edge

Reasoning models

Bare-metal GPU racks

AI agents

Code generation

Cost-optimized serving

Where each one is stronger

Bare-metal first

Engine breadth

Energy-aware routing

Heterogeneous fleets

CI-gated targets

Production deployment profiles

One command to install

Built in the open

Contribute

Discussions

Docs

Releases

Try it on your hardware

Orchestrate LLM inference
across your fleet.