Routing: KV-aware scoring deep dive

The router's job is to pick the best node for each request in sub-millisecond p99. "Best" is a four-term linear combination of signals scraped from every live agent.

The score

For every healthy node n that hosts the requested model, we compute:

score(n) = w_kv  · overlap(prefix, n)
         + w_load · (1 − util(n))
         + w_pwr  · (1 − norm_watts(n))
         + w_cap  · capacity(n)

Each term is normalised to [0, 1]; the four weights are read live from etcd at /cognitora/routing/policy (defaults 0.55, 0.25, 0.10, 0.10) and applied via an arc_swap::ArcSwap so policy changes are zero-cost on the hot path.

overlap(prefix, n)

The router hashes the prompt's first ~64 tokens to a chain of BLAKE3 digests (one per 16-token block). For each candidate node it walks the in-memory prefix index (fed by confirmed-KV etcd keys, optimistic post-dispatch inserts, and agent-reported kv_epoch / eviction-burst pruning) to find the longest contiguous cached prefix. The fraction of matched blocks is the overlap signal. This is the single most-important term; it's why KV-aware routing exists.

Today this is local scoring only. The router does not call cgn-kvcached on the hot path; overlap comes from agent heartbeat prefix claims, Confirmed-KV etcd keys, and optimistic post-dispatch inserts. Live overlap queries against cgn-kvcached are on the v0.5 roadmap — see also vs Dynamo.

util(n)

Each cgn-agent publishes its in-flight request count and total GPU SM utilisation every [agent].heartbeat (default 5 s). The router computes util = max(in_flight / max_concurrent, sm_pct) and uses (1 − util) so a busy node loses score linearly.

norm_watts(n)

cgn-metrics exports cgn_power_watts{component=...} per host. The router fetches a 30 s rolling average and normalises by the fleet's max so an idle node with low watts wins, all else equal. This is the only term whose default weight is small (0.10); turn it up for energy-aware clusters.

capacity(n)

Static metadata: TP size, dtype, VRAM, free model slots. Mostly breaks ties when two nodes are equally busy and equally cached.

Tie-break

When two nodes score within ε (0.01 default) the router falls back to a stable hash of (node_id, prefix_hash). That keeps a prefix-bound request hitting the same node across retries; the stickiness is what makes KV-aware routing worth it under burst traffic.

Admission

After scoring, the router calls Admission::try_admit(model, role) on the HTTP gateway and gRPC Generate paths before dispatch. The admission counter is per-(model, role) and bounded by [router.admission].max_queue. A Permit is held for the lifetime of the request and decrements on drop (RAII). When the queue is full the router returns 429 immediately without calling the agent. When [router.autoscaler].deadline_admission = true, requests whose estimated TTFT exceeds their deadline are rejected after routing.

We deliberately don't queue: queueing inflates TTFT and the client's deadline budget is more useful at the source. The queue parameter is therefore a hard cap on concurrency, not a buffer.

Cascade

Optional. When [router.cascade].enabled = true and the request's model has a cascade chain configured, the router runs the request against the smallest model first. After the response comes back, the mean log-probability across emitted tokens is compared against [router.cascade].confidence_threshold. If it's below threshold the router escalates to the next model in the chain.

The score function above runs once per cascade step, so each step goes to the best-suited node for that model. The cascade FSM lives in cgn-router/src/cascade.rs; the gateway invokes it as a wrapper around routing::pick.

Disaggregation

Optional. When [router.disagg].enabled = true and the prompt exceeds colocate_below_tokens, the router asks for two nodes: a prefill agent that runs the first forward pass and emits KV blocks into cgn-kvcached, and a decode agent that picks up those blocks (via QUIC/RDMA) and streams tokens. The handshake adds one round trip but pays for itself on long-prompt / short-completion workloads because decode-only nodes can be smaller / cheaper.

The QUIC transport is the default; RDMA is available behind the rdma feature flag on Linux hosts with ibverbs.

Where the code lives