cgn-router: KV-aware request router

OpenAI-compatible HTTP gateway with BLAKE3 prefix scoring, per-tenant admission, optional cascade and disaggregated dispatch, and policy loaded from etcd without blocking the hot path.

Overview

cgn-router is the only binary that speaks HTTP to clients. It authenticates (cgn-auth), applies rate limits (cgn-ratelimit), hashes the prompt into sequence-chained BLAKE3 digests, scores every healthy agent that hosts the requested model, picks the best node, and opens an Agent.Generate gRPC stream to cgn-agent. Streaming responses are re-encoded as OpenAI SSE chunks.

For background on the score function and etcd keys, read Routing architecture and the topology overview.

Features

  • Four-term routing score: KV prefix overlap, load, power (from metrics), and capacity; weights hot-reloaded from etcd via lock-free arc_swap
  • Sequence-chained BLAKE3 digests over 32-token chunks of real tokenizer ids, salted per tenant
  • Live per-node in-flight counts and session affinity on top of the score
  • Per-(model, role) admission with per-tenant wait queues, and TTFT SLO hints
  • Retry on another node until the first token; drain on SIGTERM
  • Virtual keys: scopes for models, tenants, tokens per minute and budgets
  • Optional cascade (SLM → larger model on low confidence) and disaggregated routing (prefill vs decode pools)
  • Cross-cluster federation hooks (forwarding path documented in Protocols)
  • API key and OIDC auth; optional mTLS on gRPC paths

Architecture

Client → :8080 HTTP → auth → rate limit → scopes + limits → tokenize → admission → prefix digests → score candidates → agent gRPC → first token → SSE stream.

Admin and Prometheus scrape live on listen_admin (default 0.0.0.0:9091): /healthz, /readyz, /metrics. /readyz returns 503 while draining, or when discovery is configured and no node is ready.

Configuration (highlights)

Binary defaults (the example config sets some of these differently):

KeyTypeDefaultDescription
[router].listen_httpstring0.0.0.0:8080OpenAI HTTP API
[router].listen_grpcstring0.0.0.0:9090Router gRPC surface
[router].listen_adminstring0.0.0.0:9091Health + Prometheus
[router].drain_timeoutduration30sIn-flight time after SIGTERM
[router].first_token_timeoutduration300sRetry elsewhere after this silence
[router].max_body_mbu3232HTTP body limit
[router].kv_tenant_isolationbooltrueSalt prefix digests per tenant
[router.score_weights].*f32kv 0.55, load 0.25, power 0.10, capacity 0.10, session 0.25Routing weights
[router.admission].*mixedmax_queue 1024, ttft_slo 800ms, wait_queue_per_tenant 64, wait_timeout 10sBack-pressure
[router.rate_limit].*mixedrps 50, burst 200, no trusted proxiesPer-principal requests per second; always on
[router.cascade].*bool / f64off by defaultMulti-model escalation
[router.disagg].*bool / u32off by defaultPrefill/decode split

[auth], [cluster], and [models.<name>] sections are shared with other binaries; see Configuration reference.

Example

[cluster]
name           = "demo"
etcd_endpoints = ["http://127.0.0.1:2379"]

[security]
require_mtls = true

[auth]
enabled       = true
api_keys_file = "/etc/cognitora/keys.txt"

[router]
listen_http  = "0.0.0.0:8080"
listen_grpc  = "0.0.0.0:9090"
listen_admin = "127.0.0.1:9091"

[router.score_weights]
kv       = 0.55
load     = 0.25
power    = 0.10
capacity = 0.10

# vLLM treats the model name as the Hugging Face repo id.
[models."meta-llama/Meta-Llama-3-70B-Instruct"]
tp        = 4
tokenizer = "/models/llama3-70b/tokenizer.json"

Dependencies

  • etcd: node registry and routing policy
  • cgn-agent: gRPC streaming backends
  • cgn-metrics (optional but recommended): power term for scoring

Operational targets

Latency budgets and behaviour are summarized in SLOs. Symptom-led debugging: Runbook: Router down.

Related documentation

Source: rust/services/cgn-router/