cognitora
Open-source · Apache-2.0

Orchestrate LLM inference
across your entire fleet.

The open-source inference orchestration layer. Engine-agnostic, KV-aware, and Kubernetes-native — from datacenter GPUs to Apple Silicon (MLX), built in Rust.

Get Started GitHub Docs
RustOpenAI APIvLLMSGLangllama.cppMLXcgn-inferTensorRT-LLMKubernetesQUIC/RDMA

Overview

The inference orchestration layer

Cognitora doesn't replace your inference engine — it coordinates multiple engines into a unified, distributed cluster.

Sub-millisecond routing

Sequence-chained BLAKE3 digests with longest-prefix overlap. Under 500µs per routing decision.

Engine-agnostic

Orchestrates vLLM, SGLang, TensorRT-LLM, llama.cpp, Apple MLX (mlx-lm), and the first-party cgn-infer engine through a single OpenAI-compatible gateway.

Production-hardened

mTLS everywhere. Cosign-verified releases. OIDC, API-key, and RBAC out of the box.

1 to 1,000+ GPUs

Cross-datacenter federation via QUIC. Scale from a single node to a global fleet.

Platform

Modular by design

Each component is independently deployable. Run the full stack on Kubernetes or pick the pieces you need.

cgn-router

KV-aware request router

OpenAI-compatible gateway. BLAKE3 sequence digests + longest-prefix overlap for KV-aware routing. Disaggregates prefill from decode.

Learn more
cgn-agent

Engine supervisor

Per-node agent driving any OpenAI HTTP surface. Built-in drivers for vLLM, SGLang, llama.cpp, MLX, the native cgn-infer engine, and more. NVML telemetry, KV handoff via NixlConnector, multi-engine clusters.

Learn more
cgn-kvcached

Multi-tier KV cache

GPU/RAM/SSD tiered cache with cross-node QUIC/RDMA fetch. Pluggable offload: nixl, lmcache, hicache, or kvbm.

Learn more
cgn-metrics

Power-aware telemetry

Prometheus aggregator with Redfish, IPMI, and DCGM. Power-aware routing deprioritizes thermally stressed nodes.

Learn more
cgn-ctl

Admin CLI

Embeds Helm. Cluster bring-up, model deploys, traffic shaping, debugging. Same CRDs the operator manages.

Learn more
cgn-operator

Kubernetes operator

kube-rs. CRDs for InferenceCluster, ModelPool, RoutingPolicy. Declarative model rollouts and traffic splits.

Learn more
cgn-inferpreview

Native inference engine

First-party GGUF engine on Candle (CPU/Metal/CUDA). Runs models directly — no external server — behind the same OpenAI HTTP surface. Continuous batching and layer-pipeline distribution.

Learn more

Features

Built for datacenter GPUs and Apple Silicon

KV-Aware Routing

Sequence-chained BLAKE3 digests eliminate redundant prefill across the cluster.

Prefill/Decode Split

Separate replica pools via NixlConnector handoff. Tune hardware per phase.

Multi-Tier KV Cache

RAM (DashMap), SSD (RocksDB), and cross-node QUIC peer fetch.

Energy-Aware Scheduling

Redfish/IPMI/DCGM telemetry. Deprioritize thermally stressed nodes.

Cross-Cluster Federation

Router forwards across clusters. KV cache peers via QUIC.

Multi-Model Cascade

SLM→LLM gating on log-probability. Save costs, preserve quality.

Pluggable KV Offload

One TOML knob: none, nixl, lmcache, hicache, or kvbm.

Engine-Agnostic Agent

Drives any process exposing the OpenAI HTTP surface (chat, completions, models, health).

Apple Silicon (MLX)

engine.kind = mlx spawns mlx-lm’s OpenAI-compatible server. Example: examples/apple-mlx/.

Native Engine (cgn-infer)

First-party GGUF engine on Candle — run models directly, no external server. engine.kind = cgn_infer. Preview.

Compatibility

Engine support matrix

Any engine exposing the OpenAI HTTP surface. Feature depth varies by backend.

FeaturevLLMSGLangllama.cppMLXTRT-LLMcgn-inferpreviewOpenAI
OpenAI HTTP gateway
KV-aware routing
Prefill/decode disaggregate————
KV offload — LMCache
KV offload — HiCache
KV offload — KVBM
Multi-tier KV (RAM/SSD)
Multi-model cascade
Energy-aware admission

wired through config, awaiting upstream support · preview = cgn-infer, the experimental first-party native engine (Llama-family GGUF)

Architecture

Disaggregated by design

Prefill and decode on separate pools. KV-aware routing places requests where caches live. Internal gRPC with mTLS everywhere.

CONTROL PLANEcgn-operatorCRDscgn-ctlCLIetcdstate + policiesDATA PLANEClientOpenAI SDKHTTPcgn-routerBLAKE3 KV routingDisaggregated dispatch:8080 HTTP · :9090 gRPCgRPCcgn-agent ×NprefillvLLM / SGLang / MLX / cgn-inferNixlcgn-agent ×NdecodeToken streaming · NVMLUDScgn-kvcachedGPU → RAM → SSDDashMap · RocksDBCross-node QUIC fetch:7090 gRPC · :7091 QUICSSE streamOBSERVABILITYcgn-metricsRedfish · DCGMPrometheusGrafanapower score

Bare Metal

systemd on Linux

Kubernetes

Helm + operator CRDs

Cloud

Terraform: AWS, GCP, Azure, Hetzner

Local Dev

Docker, Ollama, MLX on macOS

Use Cases

From single models to fleet orchestration

Apple Silicon & Edge

Run the same router stack locally with MLX (mlx-lm): the agent spawns an OpenAI-compatible server on Apple Silicon while KV-aware routing stays consistent.

MLXmacOSDev laptop

Reasoning Models

MoE and chain-of-thought workloads benefit from disaggregated prefill/decode. Split compute phases across specialized pools.

MoECoTLong context

Kubernetes AI Scaling

Deploy across GKE, EKS, AKS with Helm and a native operator. CRDs for InferenceCluster, ModelPool, RoutingPolicy.

HelmCRDsMulti-cloud

AI Agents

Agent workloads reuse context heavily. KV-aware routing lands follow-ups where the prefix cache lives.

KV cachePrefix reuseLow TTFT

Code Generation

Large repo contexts cached across RAM, SSD, and peer nodes. Hot contexts stay resident, cold ones page transparently.

Context reuseMulti-tierStreaming

Cost-Optimized Serving

SLM→LLM cascade gates on log-probability. Cut costs on straightforward queries without sacrificing quality.

SLM→LLMCostQuality gates

Performance

CI-gated targets

Every merge to main is gated on these numbers. Harnesses in the repo.

Routing decision

< 500µs

per vCPU, CI-gated

HTTP overhead

< 3ms

p99 vs direct engine

Warm cache hit

< 200µs

RAM tier lookup

Cold cache hit

< 5ms

SSD/RocksDB tier

Cross-node fetch

< 12ms

1 MiB, QUIC, 10 GbE

Cache hit ratio

≥ 55%

representative traces

Energy efficiency

1.4×

vs round-robin

Model bring-up

< 30s

using recipes

Recipes

Production deployment profiles

model/engine/topology/ — 3-line up.sh, serving in <30s.

ModelEngineTopology
Llama 3.2 3B (MLX)MLX (mlx-lm)Apple Silicon example
Llama-3.1 8BvLLM1×A100 aggregated
vLLM1×A100 disaggregated
SGLang1×A100
vLLM + LMCache1×A100
cgn-inferNative GGUF single-node
Llama-3.3 70B FP8vLLMTP=4 multi-GPU
Qwen3-7BvLLM1×GPU
SGLang1×GPU
DeepSeek-v4-FlashvLLMMulti-GPU
SGLangMulti-GPU
$ bash recipes/llama3-8b/vllm/agg/up.sh

Quickstart

One command to install

$curl -fsSL https://inference.cognitora.dev/install | sh
Linux
# prebuilt binaries (x86_64 / aarch64; default = latest GitHub release)
# inference.cognitora.dev/install redirects to deploy/installer/install.sh on GitHub
curl -fsSL https://inference.cognitora.dev/install | sh

# pin version / prefix — vars apply to the shell running install.sh (see deploy/installer/install.sh)
curl -fsSL https://inference.cognitora.dev/install | CGN_VERSION=v0.4.0 sh
curl -fsSL https://inference.cognitora.dev/install | CGN_PREFIX=$HOME/.local sh

# verify
cgn-ctl version
NEXT — Send a request (Linux)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

Community

Built in the open

Contribute

Issues, PRs, design discussions. Apache-2.0, CLA-free.

Open

Discussions

Architecture proposals, deployment questions, roadmap.

Open

Docs

Guides, CRD references, runbooks — live from GitHub.

Open

Releases

Changelogs, migration notes. Cosign-verified artifacts.

Open

Ready to deploy?

Star the repo, follow the quickstart, bring distributed inference to your fleet.

Star on GitHubQuickstart