cgn-kvcached

Multi-Tier KV Cache Daemon

GPU/RAM/SSD tiered KV cache with cross-node QUIC/RDMA fetch. The shared memory layer that makes KV-aware routing possible.

Overview

cgn-kvcached manages a three-tier KV block store: GPU HBM (engine-internal), RAM (warm, DashMap with approximate-LRU), and SSD (cold, RocksDB-indexed with io_uring). When a block is not found locally, it fetches from peer nodes via QUIC (1-RTT, 0-RTT on repeat connections) or optional RDMA (GPUDirect). Block addressing uses a 36-byte key: BLAKE3-256 of (model, dtype, prefix tokens) plus layer index.

Features

Three-tier storage: GPU HBM → RAM (DashMap, approximate-LRU) → SSD (RocksDB, io_uring)
36-byte block addressing: BLAKE3-256(model, dtype, prefix) + layer index
Cross-node QUIC fetch: 1-RTT connection, 0-RTT on repeat peers, multi-stream multiplexing
Optional RDMA support via GPUDirect with identical frame codec
Binary frame format: Frame { addr, model, layer, bytes }
RocksDB column-family (cf=kv) for persistent block metadata index
In-memory DashMap fallback for development builds
Non-blocking eviction: never blocks writes to make room
Background eviction loop (1 Hz): RAM watermark spill to SSD + SSD TTL expiry
Approximate-LRU victim selection via per-block touch timestamps, O(1) occupancy tracking
Live cache observability: hit/miss/eviction/spill counters and bytes pushed/pulled via Kv.Stats
Promote RPC reads spilled blocks back from SSD into RAM
SSD files stored as <short(digest)>-<layer>.kvb with O_DIRECT
Survives restarts via RocksDB index reconciliation

Architecture

Engine evicts GPU block → cgn-kvcached receives via UDS → stores in RAM tier (DashMap) → a 1 Hz background loop spills the coldest blocks to SSD when RAM crosses the high watermark and TTL-expires stale SSD blocks. On cache miss: local RAM → local SSD (promoted back to RAM on read) → QUIC peer fetch → engine re-prefill. Metadata index in RocksDB is authoritative for warm/cold tiers. Exposes cognitora.v1.Kv gRPC service (Lookup/Push/Pull/Promote/Stats) with live hit/miss/eviction counters.

Configuration

Key	Type	Default	Description
kv.listen	string	0.0.0.0:7090	gRPC listen address
kv.uds	string	""	Unix domain socket path for agent communication
kv.ram_gib	u32	4	RAM tier capacity in GiB
kv.ssd_dir	string	"/var/lib/cognitora/kv/ssd/"	SSD storage directory
kv.ssd_gib	u32	64	SSD tier capacity in GiB
kv.ssd_ttl_secs	u64	86400	Time-to-live for SSD-cached blocks in seconds (0 disables)
kv.ram_high_watermark	f32	0.90	RAM occupancy fraction that triggers spill of coldest blocks to SSD
kv.evict_interval_ms	u64	1000	Background eviction loop interval in milliseconds
kv.index_dir	string	"/var/lib/cognitora/kv/index/"	RocksDB index directory
kv.transport	enum	"quic"	Cross-node transport: quic or rdma
kv.quic_listen	string	0.0.0.0:7091	QUIC listen address for peer fetch
kv.block_size_tokens	u32	16	Tokens per KV block

Example

toml

[cluster]
name           = "production"
state_backend  = "etcd"
etcd_endpoints = ["http://etcd-0:2379"]

[security]
require_mtls = true

[kv]
listen            = "0.0.0.0:7090"
uds               = "/tmp/cognitora-kv.sock"
ram_gib           = 8
ssd_dir           = "/var/lib/cognitora/kv/ssd"
ssd_gib           = 128
index_dir         = "/var/lib/cognitora/kv/index"
transport         = "quic"
quic_listen       = "0.0.0.0:7091"
block_size_tokens = 16
ssd_ttl_secs       = 86400  # expire cold blocks after 24h
ram_high_watermark = 0.90   # spill coldest blocks above 90% RAM
evict_interval_ms  = 1000   # eviction loop cadence

Performance

GPU tier lookup: < 30µs

RAM tier (warm) hit: < 200µs

SSD tier (cold) hit: < 5ms

Cross-node QUIC fetch (1 MiB, 10 GbE): < 12ms

Cache hit ratio target: ≥ 55% on representative traces

Background eviction: 1 Hz LRU walk, never blocks writes

cgn-agent

cgn-metrics