One control plane for every engine

Orchestrate LLM inference
across your entire fleet.

The open-source inference orchestration layer. Engine-agnostic, KV-aware, and bare-metal-first. Built in Rust, it runs the same way on a few DGX Sparks, AMD boxes, or Mac Studios on your desk as on a Kubernetes or cloud fleet — one curl install, six binaries.

Get Started GitHub Docs

RustOpenAI APIvLLMSGLangllama.cppMLXcgn-inferTRT-LLM driverBare metalKubernetesQUIC

Overview

The inference orchestration layer

Cognitora doesn't replace your inference engine. It coordinates multiple engines into a unified, distributed cluster. A drop-in alternative to orchestrators like NVIDIA Dynamo, with broader engine coverage (llama.cpp, Apple MLX, any OpenAI-compatible server), bare-metal-first deployment, and energy-aware scheduling. No Python control plane required.

Sub-millisecond routing

Sequence-chained BLAKE3 digests with longest-prefix overlap. Under 500µs per routing decision.

Engine-agnostic

Orchestrates vLLM, SGLang, llama.cpp, Apple MLX (mlx-lm), a TRT-LLM spawn driver (trtllm-serve; no default recipe or GPU E2E gate yet), and the first-party cgn-infer engine through a single OpenAI-compatible gateway.

Production-hardened

mTLS everywhere. Cosign-verified releases. OIDC, API-key, and RBAC out of the box.

Bare metal to fleet

Start on one node with systemd and six static binaries. Federation via QUIC when you outgrow a single rack.

Platform

Modular by design

Each component is independently deployable. Install six static binaries on bare metal with systemd, or add the operator when you need Kubernetes.

cgn-router

KV-aware request router

OpenAI-compatible gateway. BLAKE3 sequence digests + longest-prefix overlap from agent heartbeat claims (live cgn-kvcached queries on the roadmap). Disaggregates prefill from decode.

Learn more

cgn-agent

Engine supervisor

Per-node agent driving any OpenAI HTTP surface. Built-in drivers for vLLM, SGLang, llama.cpp, MLX, the native cgn-infer engine, and more. NVML telemetry, KV handoff via NixlConnector, multi-engine clusters.

Learn more

cgn-kvcached

Multi-tier KV cache

RAM/SSD tiered cache behind engine-owned GPU HBM, with cross-node QUIC fetch. Pluggable offload: nixl, lmcache (in-process or LMCache MP server), hicache, or kvbm.

Learn more

cgn-metrics

Power-aware telemetry

Prometheus aggregator with Redfish and NVML power telemetry. Power-aware routing deprioritizes thermally stressed nodes.

Learn more

cgn-ctl

Admin CLI

Cluster bring-up, model deploys, traffic shaping, and debugging on bare metal. Optional Helm charts when you deploy on Kubernetes.

Learn more

cgn-operator

Kubernetes operator (optional)

kube-rs operator with CRDs for InferenceCluster, ModelPool, and RoutingPolicy. Add when you need declarative rollouts on Kubernetes — not required for bare-metal installs.

Learn more

cgn-inferpreview

Native inference engine

First-party GGUF engine on Candle (CPU/Metal/CUDA). Runs models directly, with no external server, behind the same OpenAI HTTP surface. Continuous batching and layer-pipeline distribution.

Learn more

Features

From a local AI cluster to a datacenter fleet

KV-Aware Routing

Sequence-chained BLAKE3 digests + longest-prefix overlap from agent heartbeat claims eliminate redundant prefill. Completion-confirmed cache claims keep the index honest; live cgn-kvcached overlap queries are on the roadmap.

Tool Calling & Structured Output

tools, tool_choice, and response_format pass through to the engine — streaming tool-call deltas included.

Multimodal Inputs

OpenAI image content parts flow straight to vision-capable engines. Prefix hashing stays text-only.

Prefill/Decode Split

Separate replica pools via NixlConnector handoff. Tune hardware per phase.

Multi-Tier KV Cache

RAM (DashMap), SSD (RocksDB), and cross-node QUIC peer fetch.

Energy-Aware Scheduling

Redfish + NVML (rocm-smi on AMD) power telemetry in the routing score, with soft per-node watt caps. Closed-loop autoscaler drains hot nodes; SLA planner scales decode replicas from queue-depth SLOs.

Cross-Cluster Federation

Router forwards to the lowest-latency reachable peer cluster when no local node serves a model. KV cache peers via QUIC.

Gossip Discovery — No etcd

state_backend = gossip forms a UDP scuttlebutt mesh with phi-accrual failure detection. Multi-node clusters with zero external services.

Multi-Model Cascade

SLM→LLM confidence gating on buffered and streaming requests. Save costs, preserve quality.

Pluggable KV Offload

One TOML knob: none, nixl, lmcache (in-process or a node-local MP server), hicache, or kvbm.

Engine-Agnostic Agent

Spawns and supervises vLLM, SGLang, llama.cpp, MLX, a TRT-LLM spawn driver (trtllm-serve; no default recipe or GPU E2E gate yet), or any OpenAI-compatible server — with live queue/KV telemetry feeding the router.

Apple Silicon (MLX)

engine.kind = mlx spawns mlx-lm’s OpenAI-compatible server. Example: examples/apple-mlx/.

Cluster Dashboard

Standalone zero-dependency web dashboard: live req/s, tokens/s, latency and TTFT percentiles, per-node status, KV utilization, power, and J/token — straight from the Prometheus endpoints.

Native Engine (cgn-infer)

First-party GGUF engine on Candle — run models directly, no external server. engine.kind = cgn_infer. Preview.

Observability

See the whole fleet at a glance

A standalone, zero-dependency dashboard ships in the repo: one static HTML file that reads the Prometheus /metrics endpoints directly. Live requests/s, tokens/s, latency and TTFT percentiles, queue depth, per-node status, KV-cache utilization, fleet power, and energy per token. No Grafana, no backend, nothing to install.

Cognitora cluster dashboard monitoring a 16-node mixed fleet (H100, H200, A100, MI300X, L40S, A10): live requests per second, tokens per second, latency and TTFT percentiles, queue depth, fleet power, KV cache utilization, and energy per token

A 16-node mixed fleet (H100, H200, A100, MI300X, L40S, A10) serving five models under load.

Compatibility

Engine support matrix

Any engine exposing the OpenAI HTTP surface. Feature depth varies by backend. TRT-LLM is a spawn driver (trtllm-serve) with no default recipe or GPU E2E gate yet.

Feature	llama.cpp	MLX	cgn-inferpreview	OpenAI
OpenAI HTTP gateway
KV-aware routing
Prefill/decode disaggregate	—	—	—	—
KV offload — LMCache
KV offload — LMCache MP
KV offload — HiCache
KV offload — KVBM
Multi-tier KV (RAM/SSD)
Multi-model cascade
Energy-aware admission

wired through config, awaiting upstream support. LMCache MP is the standalone lmcache server plus LMCacheMPConnector (vLLM). SGLang uses --enable-lmcache. · preview = cgn-infer, the experimental first-party native engine (Llama-family GGUF)

Architecture

Disaggregated by design

Prefill and decode on separate pools. KV-aware routing places requests where caches live. Internal gRPC with mTLS everywhere.

Bare Metal

systemd on Linux

Kubernetes

Helm + operator CRDs

Cloud

Terraform: AWS, GCP, Azure, Hetzner

Local Dev

Docker, Ollama, MLX on macOS

Use Cases

From a single GPU rack to fleet orchestration

Apple Silicon & Edge

Run the same router stack locally with MLX (mlx-lm): the agent spawns an OpenAI-compatible server on Apple Silicon while KV-aware routing stays consistent.

MLXmacOSDev laptop

Reasoning Models

MoE and chain-of-thought workloads benefit from disaggregated prefill/decode. Split compute phases across specialized pools.

MoECoTLong context

Bare-Metal GPU Racks

One curl install, six static binaries, systemd units. Run Llama on a DGX Spark or a mixed AMD/NVIDIA rack without Kubernetes.

systemdBare metalMixed GPUs

AI Agents

Agent workloads reuse context heavily. KV-aware routing lands follow-ups where the prefix cache lives.

KV cachePrefix reuseLow TTFT

Code Generation

Large repo contexts cached across RAM, SSD, and peer nodes. Hot contexts stay resident, cold ones page transparently.

Context reuseMulti-tierStreaming

Cost-Optimized Serving

SLM→LLM cascade gates on log-probability. Cut costs on straightforward queries without sacrificing quality.

SLM→LLMCostQuality gates

vs NVIDIA Dynamo

Positioning

Same orchestration shape, different runtime and deployment surface. Cognitora leads on heterogeneous, energy-aware, ops-light inference — not NVL72 scale.

Bare-metal first

One curl install, six static binaries, systemd units. Kubernetes and Terraform are optional targets, not prerequisites.

Engine breadth

vLLM, SGLang, llama.cpp, MLX, OpenAI-compat, and a TRT-LLM spawn driver — mixed engines in one cluster.

Energy-aware routing

Redfish + NVML power telemetry in the routing score. Soft watt caps and autoscaler drain hints.

Heterogeneous fleets

Per-model GPU constraints route 70B off the L40S without separate deployments per hardware class.

Where Dynamo leads today

Disaggregated multimodal & video pipelines
Native KVBM tiered block manager (G1–G4)
ModelExpress GPU-to-GPU weight streaming
Grove NVL72 topology-aware gang scheduling
In-flight request migration
AIConfigurator deployment search

Credibility gaps we're closing (v0.5)

Router ↔ cgn-kvcached overlap

local heartbeat scoring today

In-flight migration

pre-token retry only

Multi-node GPU E2E

outside default CI

RDMA transport

stub behind feature gate

Full comparison doc

Performance

CI-gated targets

Every merge to main is gated on these numbers. Harnesses in the repo.

Routing decision

< 500µs

per vCPU, CI-gated

HTTP overhead

< 3ms

p99 vs direct engine

Warm cache hit

< 200µs

RAM tier lookup

Cold cache hit

< 5ms

SSD/RocksDB tier

Cross-node fetch

< 12ms

1 MiB, QUIC, 10 GbE

Cache hit ratio

≥ 55%

representative traces

Energy efficiency

1.4×

vs round-robin

Model bring-up

< 30s

using recipes

Recipes

Production deployment profiles

model/engine/topology/: a 3-line up.sh, serving in <30s.

Model	Engine	Topology
Llama 3.2 3B (MLX)	MLX (mlx-lm)	Apple Silicon example
Llama-3.1 8B	vLLM	1×A100 aggregated
	vLLM	1×A100 disaggregated
	SGLang	1×A100
	vLLM + LMCache	1×A100 in-process
	vLLM + LMCache MP	1×A100 node-local server
	cgn-infer	Native GGUF single-node
Llama-3.3 70B FP8	vLLM	TP=4 multi-GPU
Qwen3-7B	vLLM	1×GPU
	SGLang	1×GPU
DeepSeek-v4-Flash	vLLM	Multi-GPU
	SGLang	Multi-GPU

$ bash recipes/llama3-8b/vllm/agg/up.sh

Quickstart

One command to install

$curl -fsSL https://inference.cognitora.dev/install | sh

Linux

# prebuilt binaries (x86_64 / aarch64; default = latest GitHub release)
# inference.cognitora.dev/install redirects to deploy/installer/install.sh on GitHub
curl -fsSL https://inference.cognitora.dev/install | sh

# pin version / prefix: vars apply to the shell running install.sh (see deploy/installer/install.sh)
curl -fsSL https://inference.cognitora.dev/install | CGN_VERSION=v0.9.2 sh
curl -fsSL https://inference.cognitora.dev/install | CGN_PREFIX=$HOME/.local sh

# verify
cgn-ctl version

NEXT: Send a request (Linux)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

Community

Ready to deploy?

Star the repo, follow the quickstart, bring distributed inference to your fleet.

Star on GitHub Quickstart

Orchestrate LLM inferenceacross your entire fleet.

The inference orchestration layer

Sub-millisecond routing

Engine-agnostic

Production-hardened

Bare metal to fleet

Modular by design

KV-aware request router

Engine supervisor

Multi-tier KV cache

Power-aware telemetry

Admin CLI

Kubernetes operator (optional)

Native inference engine

From a local AI cluster to a datacenter fleet

KV-Aware Routing

Tool Calling & Structured Output

Multimodal Inputs

Prefill/Decode Split

Multi-Tier KV Cache

Energy-Aware Scheduling

Cross-Cluster Federation

Gossip Discovery — No etcd

Multi-Model Cascade

Pluggable KV Offload

Engine-Agnostic Agent

Apple Silicon (MLX)

Cluster Dashboard

Native Engine (cgn-infer)

See the whole fleet at a glance

Engine support matrix

Disaggregated by design

From a single GPU rack to fleet orchestration

Apple Silicon & Edge

Reasoning Models

Bare-Metal GPU Racks

AI Agents

Code Generation

Cost-Optimized Serving

Positioning

Bare-metal first

Engine breadth

Energy-aware routing

Heterogeneous fleets

CI-gated targets

Production deployment profiles

One command to install

Built in the open

Contribute

Discussions

Docs

Releases

Ready to deploy?

Orchestrate LLM inference
across your entire fleet.