cognitora
All components
cgn-infer

Native Inference Engine

First-party GGUF inference engine built on Candle (Rust). Serves the OpenAI HTTP surface directly — no external engine required. Experimental / preview.

Overview

cgn-infer is Cognitora's own inference engine — the seventh binary. Where the other components orchestrate external engines like vLLM, SGLang, and llama.cpp, cgn-infer loads quantized GGUF models itself via mmap and runs them on Candle's CPU, Metal, or CUDA backends. It exposes the same OpenAI HTTP/SSE surface as every other engine, so cgn-agent spawns and supervises it exactly like vLLM. Phase 1 covers Llama-family GGUF models with sequential request serving; continuous batching and distributed layer-pipeline inference are on the roadmap.

Features

  • GGUF model loading via mmap: page-cache resident, no heap copy, llama.cpp-compatible model files
  • Candle tensor backends: CPU, Metal (Apple Silicon), and CUDA behind a swappable Runtime trait
  • OpenAI-compatible HTTP: /v1/chat/completions, /v1/completions, /v1/models, /healthz with SSE streaming
  • Standard sampling: temperature, top-p, top-k, repetition penalty
  • Paged KV cache with block-hash prefix reuse, aligned with cgn-kvcached's BLAKE3 block scheme
  • Drop-in engine under cgn-agent: engine.kind = "cgn_infer" renders the launch argv automatically
  • Roadmap: continuous batching scheduler (prefill chunking + batched decode)
  • Roadmap: distributed layer-pipeline parallelism over gRPC (mTLS) with etcd discovery
  • Roadmap: more architectures (Qwen, Mistral, MoE) and quant formats

Architecture

cgn-agent spawns `cgn-infer serve --model <gguf> --host <h> --port <p> --ctx <n> --threads <n>` and supervises it like any other engine. Inside: axum OpenAI API → scheduler → Runtime trait → Candle backend (CPU/Metal/CUDA) with a paged KV cache and prefix reuse. In pipeline mode (planned), a coordinator streams hidden states to layer-range workers over tonic gRPC.

Configuration

KeyTypeDefaultDescription
engine.kindenum"vllm"Set to "cgn_infer" to use the native engine
engine.cgn_infer.binarystring"cgn-infer"Path or PATH-name of the cgn-infer binary
engine.cgn_infer.hoststring"127.0.0.1"Where the engine listens
engine.cgn_infer.portu168001Engine port; must match engine.url
engine.cgn_infer.ctxu324096Context window
engine.cgn_infer.threadsu324CPU thread count
engine.kv_offloadenum"none"Only "none" is valid for cgn-infer (Phase 1)

Example

toml
[agent]
listen  = "0.0.0.0:7080"
node_id = "agent-mac-01"

[engine]
kind       = "cgn_infer"
url        = "http://127.0.0.1:8001"
kv_offload = "none"

[engine.cgn_infer]
port    = 8001
ctx     = 8192
threads = 8

[models."llama-3.1-8b"]
path = "/models/llama-3.1-8b-q4_k_m.gguf"

Performance

Phase 1: Llama-family GGUF only, sequential request serving

Zero-copy weight loading via mmap (page-cache resident)

Kernels via Candle (CPU / Metal / CUDA), swappable behind the Runtime trait

Previous

cgn-operator