Cognitora vs NVIDIA Dynamo

This page is an axis-by-axis comparison between Cognitora and NVIDIA Dynamo, the closest peer in the open-source distributed-inference space.

It is not a marketing pitch: where Dynamo is ahead, we say so; where we are ahead, we explain why; where the projects converge, we say that too. The goal is to help an operator decide which stack matches their constraints.

For the lower-level "how does Cognitora compose engines, offload backends, and the cross-cluster index" deep dive, see kv-strategy.md. For the routing math, see routing.md. For Cognitora's own architecture, see ../ARCHITECTURE.md.

TL;DR

• Same shape. Both projects are orchestration layers above inference engines. Both ship KV-aware routing, disaggregated prefill/decode, and multi-tier KV caching. Both are Apache-2.0.
• Different runtime. Dynamo is Rust core + Python frontend, designed Kubernetes-first with an operator and CRDs. Cognitora is six static binaries, designed bare-metal-first with the same artifacts running under systemd, Helm, or Terraform.
• Different engine breadth. Dynamo treats vLLM / SGLang / TRT-LLM as the universe. Cognitora adds llama.cpp and any OpenAI-compatible process (Ollama, hosted endpoints, sidecars) as first-class drivers.
• Different KV story. Dynamo ships KVBM as the in-house tiered block manager. Cognitora doesn't ship its own; it integrates LMCache, SGLang HiCache, and KVBM as alternatives behind one TOML knob, plus its own cgn-kvcached cross-cluster index on top.
• Pick Dynamo if you want NVIDIA-aligned reference deployments, video pipelines / disaggregated multimodal, or NVL72 gang scheduling.
• Pick Cognitora if you want pure-binary deployment, bare-metal / hybrid topologies, energy-aware admission, llama.cpp at the edge, or cross-cluster federation.

Credibility gaps (v0.5)

Gaps that would undermine a Dynamo comparison if left unaddressed. See plan.md for the matching roadmap items.

Already shipped (previously listed as gaps in internal reviews):

• Streaming cascade — stream_run_cascade in the gateway.
• Live engine telemetry — queue depth and KV occupancy in the agent heartbeat.
• Closed autoscaler loop — drain hints + operator replica scaling.

Side-by-side: capabilities

The vertical groupings are what an operator typically asks about, not the internal module names of either project.

Routing

Engines

KV cache

This is the area where the projects differ the most. Both stacks recognise the same four layers; they own and integrate them differently.

The pragmatic upshot: Dynamo owns the offload data plane via KVBM; Cognitora delegates that to the engine's preferred connector (LMCache, HiCache, KVBM) and focuses on the cross-cluster index that sits above all of them.

Disaggregated serving

Operator / control plane

Autoscaling and topology

Modalities

Fault tolerance

Observability

What we have that Dynamo doesn't

Differentiators where Cognitora is currently ahead:

1. Engine breadth. llama.cpp + OpenAI-compatible engines are first-class drivers, not adapters. This means the same control plane runs on a laptop, a CPU edge box, an NVIDIA H100 or AMD MI300X cluster, and an Ollama-backed dev sandbox.
2. Bare-metal-first deployment. deploy/systemd/ units, a one-line installer with cosign-verified release tarballs, and Terraform recipes for the four major clouds, without requiring Kubernetes.
3. Pure-binary runtime. Six static binaries, no Python control plane, no JVM, no operator install required. The same artifacts work under systemd, Helm, or kubectl run.
4. Sequence-chained KV digests. Routing scores are computed on prefixes that encode position, so two requests with identical chunks in different orders never share routing fate.
5. Cross-cluster federation. cgn-router::federation forwards across clusters; cgn-kvcached peers across QUIC. Multi-region inference doesn't need a Kubernetes-of-Kubernetes.
6. Energy-aware admission. cgn-power reads Redfish + NVML (or rocm-smi on AMD) and feeds into the router scoring weight; a soft per-node watt cap ([agent].watt_limit) steers traffic to under-cap nodes, and the autoscaler drains nodes that hit thermal or power caps while the operator closes the loop. [carbon] adds grid-intensity polling and defers low-priority HTTP requests during high-carbon periods.
7. Multi-model SLM → LLM cascade. cascade::Cascade::run runs the cheap model first and only escalates when the log-probability of the cheap answer falls below threshold.
8. Single TOML knob for KV offload. engine.kv_offload swaps none / nixl / lmcache / hicache / kvbm without editing the engine argv yourself.
9. Capability-aware routing for mixed fleets. Agents publish GPU identity (name, vendor, VRAM); per-model min_vram_mb / require_gpu constraints keep a 70B model off the L40S in the corner without separate deployments per hardware class.

What Dynamo has that we don't yet

Areas where Dynamo is currently ahead:

1. Disaggregated multimodal & video pipelines. Dynamo ships disaggregated encode/prefill/decode for images and native FastVideo + SGLang Diffusion integration. We pass image inputs through to the engine (0.7) but don't disaggregate the encode stage, and we have no video story.
2. KVBM as a built-in tiered block manager. Dynamo's KVBM owns GPU + Host + Disk + Remote pools natively. We integrate KVBM as one of several offload backends, but we don't ship our own GPU-tier block pool.
3. ModelExpress. GPU-to-GPU weight streaming across NIXL/NVLink for fast cold-starts. Cognitora cold-starts use disk-bound from_pretrained paths today.
4. Grove gang scheduling. Topology-aware NVL72 placement. Our cgn-operator does basic node selection but isn't fabric-aware.
5. AIConfigurator. Workload simulator that searches thousands of deployment configs to find the optimal one. We have benchmark harnesses (scripts/bench/) but no auto-search yet.
6. In-flight request migration. When a Dynamo worker dies, in-flight requests can migrate to a healthy replica. Cognitora restarts the agent and retries idempotent requests; mid-stream migration is future work.
7. Zero-config deploy (DGDR). Specify model + SLA in one YAML and Dynamo profiles + plans + deploys. Our equivalent is the recipes/ tree: pre-baked, not generated.
8. Tool-calling toolkit. Dynamo ships a NeMo Agent Toolkit integration. Cognitora passes tool calls through to the engine but adds nothing on top.

Where the projects converge

These are areas where the two stacks made similar choices and the operator-visible behaviour is comparable:

• OpenAI-compatible HTTP gateway as the public API.
• gRPC / TCP for internal request fan-out.
• etcd for cluster state.
• NixlConnector for prefill→decode KV handoff.
• LMCache support as an engine-side offload backend.
• <model>/<engine>/<topology>/ recipe layout.
• Apache-2.0 license, OSS-first development model.

When to pick which

Migration / interop

Both stacks consume the same --kv-transfer-config JSON shapes for LMCache, KVBM, and NIXL; Cognitora's auto-renderer was modelled on the same patterns. A vLLM container that Dynamo can launch in agg or disagg mode is the same container Cognitora launches with engine.kv_offload = "lmcache" (or kvbm).

The recipe layouts are also similar: a Dynamo recipes/<model>/<engine>/<topology>/ folder maps to a Cognitora recipes/<model>/<engine>/<topology>/ folder of TOML profiles. Porting between the two is mostly mechanical.

Roadmap impact

Items on the Cognitora roadmap (plan.md) that close the deltas above:

• Multimodal text+image E/P/D: track once vLLM and SGLang ship stable disaggregated multimodal hooks.
• Native G1 GPU pool in cgn-kvcached: optional, only if a workload doesn't fit any of the L2 backends.
• WSPT prefill scheduling: Dynamo's "weighted shortest predicted task" admission. The KV-overlap signal needed for it already exists in our router; the queue restructure is what's missing.
• Federated peer-fetch policy: bound egress when peer-fetching from another cluster.
• Workload simulator: cgn-ctl bench plan that searches the recipe matrix.

Items where we deliberately don't intend to converge:

• No Kubernetes-only path. Bare metal stays first-class.
• No Python control plane. New control-plane logic stays in Rust.
• No CRD-as-config. Recipes stay flat TOML.