Configuration reference

Cognitora binaries read a single TOML file (default /etc/cognitora/cognitora.toml) plus environment overrides. The authoritative schema lives in rust/libraries/cgn-core/src/config.rs; the canonical example with every section documented inline lives at configs/cognitora.toml.example.

Per-binary narrative docs (features, ports, dependencies): see docs/platform/.

Sections

[engine] — pluggable inference engine

Cognitora's cgn-agent is engine-agnostic: any process that exposes the OpenAI HTTP surface (/v1/completions, /health, /v1/models) plugs in.

When kind = "openai_compat" the agent does not spawn a child process; it only proxies to whatever is at engine.url. Use this with systemd / Kubernetes / a sidecar that owns the engine lifecycle.

Engine selection

The supported engine kinds map to the same OpenAI HTTP surface, so they are fully interchangeable from the router's perspective:

• vllm — vllm serve <model> --tensor-parallel-size <N> .... Best general-purpose GPU engine; supports continuous batching and chunked prefill out of the box.
• sglang — python -m sglang.launch_server --model-path <model> --tp <N> .... Adds RadixAttention prefix caching that complements Cognitora's cross-node prefix routing — the router still picks the node with the longest cached prefix, and SGLang then reuses cache inside that node.
• llama_cpp — CPU-friendly fallback (and CUDA-offload via n_gpu_layers); useful for laptops, CI, and edge deployments.
• mlx — python3 -m mlx_lm.server --model <hf_or_path> --host <h> --port <p> …. Apple Silicon / macOS only (mlx-lm). Use kv_offload = "none" only.
• cgn_infer (experimental / preview) — cgn-infer serve --model <gguf> --host <h> --port <p> --ctx <n> --threads <n>. Cognitora's first-party native engine (Rust + Candle, GGUF via mmap) with continuous batching (llama/qwen2 GGUFs; qwen3/gemma3/phi3/MoE serve sequentially) and optional multi-node layer-pipeline mode via [models.*.pipeline]; only kv_offload = "none" is valid. See docs/architecture/cgn-infer.md.
• openai_compat — proxy-only.

Engine-side KV offload

engine.kv_offload selects which connector cgn-agent injects when spawning the engine. The router is unaware of this dial — it only sees prefix-overlap signals via cgn-kvcached either way — so swapping backends is a one-line change.

Disagg topologies ([agent].role = "prefill" or "decode") compose the chosen backend with NIXL automatically. The full table — including the exact JSON blobs — lives in docs/architecture/kv-strategy.md.

LMCache, HiCache, and KVBM all require the corresponding Python package to be installed in the engine's virtualenv. cgn-agent does not install them; the recipe's up.sh warns when they're missing.

Per-model knobs

[models."<name>"].path is required when engine.kind = "llama_cpp" or "cgn_infer" (the filesystem path to a .gguf file). For SGLang or MLX, path is optional: when unset the spawn argv uses the model table key as the Hugging Face repo id; when set, it is passed to --model as a local directory. vLLM behaves the same way as SGLang for path.

[models.*.pipeline] — cgn-infer layer pipeline

With engine.kind = "cgn_infer", a per-model pipeline block splits the model across processes/nodes by layer range. The agent spawns one cgn-infer worker per spawn = true member (before the coordinator), registers workers in etcd as non-servable, and restarts the whole pipeline if any member exits.

[models."llama3-8b"]
path = "/models/llama3-8b.gguf"

[models."llama3-8b".pipeline]
coordinator_layers  = "0:11"    # embedding + layers [0,11) + LM head, local
activation_encoding = "f16"     # or "int8"

[[models."llama3-8b".pipeline.workers]]
listen = "127.0.0.1:9101"       # spawned locally by this agent
layers = "11:22"

[[models."llama3-8b".pipeline.workers]]
listen   = "10.0.0.5:9101"      # managed elsewhere (another agent / systemd)
layers   = "22:32"
endpoint = "http://10.0.0.5:9101"
spawn    = false

The coordinator validates at startup that coordinator_layers plus the worker slices, in order, exactly tile the model's layer count. Pipeline mode is limited to llama/qwen2 GGUF architectures.

Legacy aliases

[agent].vllm_url and [agent].vllm_cmd from older configs still work but emit a one-time warning. Migrate them to [engine].url and [engine.vllm].extra_args respectively.

Overrides

Every TOML key has a corresponding environment variable: prepend COGNITORA__, separate sections with __, and use SCREAMING_SNAKE.

# Override [router].listen_http
COGNITORA__ROUTER__LISTEN_HTTP=0.0.0.0:8000

# Disable auth for a dev run
COGNITORA__AUTH__ENABLED=false

CLI flags take precedence over the env, which takes precedence over the TOML file, which takes precedence over compiled defaults.

Hot reload

The following keys reload without restart:

• [auth].api_keys_file (sha256 keys file is watched and re-read)
• [router.score_weights] (router subscribes to etcd /cognitora/routing/policy)
• [router.cascade] and [router.disagg] (same etcd key)

Everything else requires systemctl restart cgn-<binary> or, in K8s, a rolling restart of the corresponding deployment / DaemonSet.