Configuration reference

Cognitora binaries read a single TOML file (default /etc/cognitora/cognitora.toml) plus environment overrides. The authoritative schema lives in rust/libraries/cgn-core/src/config.rs; the canonical example with every section documented inline lives at configs/cognitora.toml.example.

Per-binary narrative docs (features, ports, dependencies): see docs/platform/.

Sections

[cluster]: state backend and discovery

[cluster] selects how nodes find each other. The default backend is etcd; setting state_backend = "gossip" switches the cluster to a UDP gossip mesh (no external services). See docs/architecture/gossip.md for the trade-offs: control-plane features such as cordon/drain, the confirmed-KV prefix feed, and routing-policy hot-reload stay etcd-only.

[engine]: pluggable inference engine

Cognitora's cgn-agent is engine-agnostic: any process that exposes the OpenAI HTTP surface (/v1/completions, /health, /v1/models) plugs in.

When kind = "openai_compat" the agent does not spawn a child process; it only proxies to whatever is at engine.url. Use this with systemd / Kubernetes / a sidecar that owns the engine lifecycle.

Engine selection

The supported engine kinds map to the same OpenAI HTTP surface, so they are fully interchangeable from the router's perspective:

• vllm: vllm serve <model> --tensor-parallel-size <N> .... Best general-purpose GPU engine; supports continuous batching and chunked prefill out of the box.
• sglang: python -m sglang.launch_server --model-path <model> --tp <N> .... Adds RadixAttention prefix caching that complements Cognitora's cross-node prefix routing; the router still picks the node with the longest cached prefix, and SGLang then reuses cache inside that node.
• llama_cpp: CPU-friendly fallback (and CUDA-offload via n_gpu_layers); useful for laptops, CI, and edge deployments.
• mlx: python3 -m mlx_lm.server --model <hf_or_path> --host <h> --port <p> …. Apple Silicon / macOS only (mlx-lm). Use kv_offload = "none" only.
• cgn_infer (experimental / preview): cgn-infer serve --model <gguf> --host <h> --port <p> --ctx <n> --threads <n>. Cognitora's first-party native engine (Rust + Candle, GGUF via mmap) with continuous batching (llama/qwen2 GGUFs; qwen3/gemma3/phi3/MoE serve sequentially) and optional multi-node layer-pipeline mode via [models.*.pipeline]; only kv_offload = "none" is valid. See docs/architecture/cgn-infer.md.
• openai_compat: proxy-only.

Engine-side KV offload

engine.kv_offload selects which connector cgn-agent injects when spawning the engine. The router is unaware of this dial (it only sees prefix-overlap signals via cgn-kvcached either way), so swapping backends is a one-line change.

Disagg topologies ([agent].role = "prefill" or "decode") compose the chosen backend with NIXL automatically. The full table, including the exact JSON blobs, lives in docs/architecture/kv-strategy.md.

LMCache, HiCache, and KVBM all require the corresponding Python package to be installed in the engine's virtualenv. cgn-agent does not install them; the recipe's up.sh warns when they're missing.

[engine.lmcache]

Used when kv_offload = "lmcache". Omitted keys keep the defaults below, so existing recipes stay on the in-process connector.

[engine.lmcache]
mode                 = "mp"          # "v1" (default) or "mp"
host                 = "localhost"   # bare host, or tcp:// / grpc://
port                 = 5555
http_port            = 8080
l1_size_gb           = 20
eviction_policy      = "LRU"
chunk_size           = 256
spawn_server         = true          # agent starts `lmcache server`
isolated_ipc         = false         # true = driver-level CUDA IPC, no hostIPC
disable_prefix_caching = false       # adds --no-enable-prefix-caching
config_file          = ""            # vLLM: operator kv-transfer-config.json
                                     # SGLang: LMCache YAML (LMCACHE_CONFIG_FILE)
connector_module_path = ""           # vLLM >= 0.20: lmcache.integration.vllm.lmcache_mp_connector
server_urls          = ""            # "tcp://a:5555,tcp://b:5555" overrides host/port

mode = "mp" also sets PYTHONHASHSEED=0 on the engine (and on the server, when the agent spawns it) unless that variable is already set. That keeps token hashes stable across processes, which both LMCache and the TensorMesh Dynamo integration require. On Kubernetes, leave spawn_server = false and use the chart's LMCache DaemonSet, or set config_file to the operator's <engine>-connection ConfigMap. isolated_ipc must match the server's --isolated-ipc flag; a mismatch fails when the two sides import each other's CUDA events.

Per-model knobs

[models."<name>"].path is required when engine.kind = "llama_cpp" or "cgn_infer" (the filesystem path to a .gguf file). For SGLang or MLX, path is optional: when unset the spawn argv uses the model table key as the Hugging Face repo id; when set, it is passed to --model as a local directory. vLLM behaves the same way as SGLang for path.

Capability constraints (heterogeneous fleets)

Per-model hardware constraints for clusters mixing GPU generations or vendors. The router filters routing candidates against the GPU identity each agent publishes in its heartbeat (NVML on NVIDIA, rocm-smi on AMD). Nodes that report no GPU identity (older agents, CPU boxes) are never filtered; the constraint only excludes nodes that affirmatively report incompatible hardware.

[models."llama3-70b"]
min_vram_mb = 80000     # only nodes reporting ≥ 80 GB of GPU memory
require_gpu = "h100"    # case-insensitive substring of GPU name or vendor

[agent].watt_limit: soft power cap

[agent]
watt_limit = 700.0   # watts; 0 (default) = uncapped

Published in the heartbeat and mirrored to cgn_cluster_node_watt_limit. When at least one candidate node is under its cap, the router routes only to under-cap nodes; when every candidate is over, routing proceeds anyway: serving beats browning out a request. Combine with the autoscaler's high_watt_threshold for drain-based enforcement.

[carbon]: grid-intensity aware admission

When enabled, the router polls a pluggable grid carbon intensity provider on a background interval and rejects low-priority OpenAI HTTP requests while the observed gCO₂/kWh exceeds intensity_threshold. Normal- and high-priority traffic is always admitted. Mark deferrable work with the X-CGN-Priority: low request header.

[carbon]
enabled = true
provider = "electricitymaps"   # static | electricitymaps | watttime
zone = "DE"
api_token = "..."              # required for remote providers
static_intensity = 300.0       # gCO2/kWh when provider = "static"
intensity_threshold = 450.0    # reject low-priority above this
poll_interval = "5m"

The latest reading is exported as cgn_carbon_intensity_gco2_per_kwh{zone}. Low-priority rejections increment cgn_router_carbon_admission_rejected_total. Until the first poll succeeds, admission fails open (requests are admitted).

[router.federation]: cross-cluster fallback

[router.federation]
enabled = true
peers = ["https://us-west.cognitora.example:7070",
         "https://eu-central.cognitora.example:7070"]

When the local cluster has no eligible node for a request's model, the gateway forwards the request to a peer cluster's router over gRPC (mTLS). Peers are probed concurrently and the lowest-connect-latency reachable peer wins. Forwarding targets the peer's gRPC surface, which only routes locally, so a request crosses at most one cluster boundary and cannot loop. Forwards are counted in cgn_router_federation_forwards_total{model,peer}.

[models.*.pipeline]: cgn-infer layer pipeline

With engine.kind = "cgn_infer", a per-model pipeline block splits the model across processes/nodes by layer range. The agent spawns one cgn-infer worker per spawn = true member (before the coordinator), registers workers in etcd as non-servable, and restarts the whole pipeline if any member exits.

[models."llama3-8b"]
path = "/models/llama3-8b.gguf"

[models."llama3-8b".pipeline]
coordinator_layers  = "0:11"    # embedding + layers [0,11) + LM head, local
activation_encoding = "f16"     # or "int8"

[[models."llama3-8b".pipeline.workers]]
listen = "127.0.0.1:9101"       # spawned locally by this agent
layers = "11:22"

[[models."llama3-8b".pipeline.workers]]
listen   = "10.0.0.5:9101"      # managed elsewhere (another agent / systemd)
layers   = "22:32"
endpoint = "http://10.0.0.5:9101"
spawn    = false

The coordinator validates at startup that coordinator_layers plus the worker slices, in order, exactly tile the model's layer count. Pipeline mode is limited to llama/qwen2 GGUF architectures.

Legacy aliases

[agent].vllm_url and [agent].vllm_cmd from older configs still work but emit a one-time warning. Migrate them to [engine].url and [engine.vllm].extra_args respectively.

Overrides

Every TOML key has a corresponding environment variable: prepend COGNITORA__, separate sections with __, and use SCREAMING_SNAKE.

# Override [router].listen_http
COGNITORA__ROUTER__LISTEN_HTTP=0.0.0.0:8000

# Disable auth for a dev run
COGNITORA__AUTH__ENABLED=false

CLI flags take precedence over the env, which takes precedence over the TOML file, which takes precedence over compiled defaults.

Hot reload

The following keys reload without restart:

• [auth].api_keys_file (sha256 keys file is watched and re-read)
• [router.score_weights] (router subscribes to etcd /cognitora/routing/policy)
• [router.cascade] and [router.disagg] (same etcd key)

Everything else requires systemctl restart cgn-<binary> or, in K8s, a rolling restart of the corresponding deployment / DaemonSet.