This page covers Bee-owned arguments and the upstream arguments whose behavior BeeLlama extends. Run llama-server --help or llama-cli --help for the full upstream surface. See BeeLlama features for use cases, limits, and measurement guidance.

KVarN values are kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, and kvarn8. K and V may use different bit widths.

CUDA, ROCm/HIP, Vulkan, and CPU consume compressed KVarN records directly in native FlashAttention paths. Vulkan requires shader Int64 and buffer-device-address support for its direct route. An explicitly supported materialization fallback retains compressed persistent storage when a native route is unavailable. Pre-Turing NVIDIA GPUs use CUDA's portable rotated-domain body-plus-tail route and require a CUDA 12.4 build or release package. CUDA 13.3 packages target Turing and newer architectures.

Draft KVarN is runtime-qualified on CUDA for draft-simple, EAGLE3, the owned MTP allowlist, DFlash1/DFlash2, and non-MLA DSpark. The CPU reference route is qualified for owned MTP. Non-causal DFlash-family models keep KVarN persistent storage but use materialized attention; the direct record-consuming route is not enabled. DSV4/MLA DSpark is incompatible with KVarN's dense K/V representation and fails closed. Shared Gemma 4 MTP and MTP architectures outside the allowlist fail closed. For Gemma 4 MTP, select target KVarN with --cache-type-k/v; the assistant reads that shared persistent cache. HIP/ROCm and Vulkan DFlash-family draft KVarN remain unqualified until backend runtime tests pass. N-gram modes do not own a KV context and reject explicit KVarN --spec-draft-type-k/v selections during argument validation.

CUDA multi-token KVarN prefill uses transient F16 K/V materialization windows. GGML_KVARN_WINDOW_CHUNK sets the positive token count per window and defaults to 65536; missing, zero, and negative values use that default, while values above the active KV length are capped to that length. A smaller value reduces peak transient scratch for concurrent long prompts but adds partial-softmax merges and changes floating-point reduction order. It does not alter context or persistent KV-cache capacity.

The KV cache precision tail (KVCPT) makes the newest attention-visible entries exact in F16 or BF16 for standard quantized and KVarN target caches. A partial tail keeps the complete quantized body and adds a compact exact-history ring. The active ubatch remains a separate graph-local exact source. A full-window SWA request uses a compact native-exact ring and omits the unread compressed SWA body. Supported draft-owned MTP and DFlash KVarN caches do not inherit the target tail or expose a draft-tail argument: their explicit request remains zero, while KVarN supplies an intrinsic exact suffix of up to 128 tokens. DFlash uses the same K/V pair for full-attention and SWA sub-caches. Other draft and auxiliary contexts remain on standard cache types.

An omitted tail type remains automatic until context placement. If the standard BF16 default lacks a complete Metal or SYCL route but F16 is complete, automatic selection warns and resolves once to F16. Explicit --kv-tail-type bf16 fails instead of changing the requested representation.

Explicit values are capped by the group's effective attention window and context capacity. KVarN values are also rounded upward to 128-token groups. Startup logs show raw, requested, effective, and window lengths, the structural group ID, participating layers, selected compact-overlay or compact-native-exact representation, actual body and exact types, logical history rows, rollback rows, graph-local body execution rows, owner backend, current-segment presence, transient estimate, and memory increments. Native routes are checked again against the final constructed operation. A mismatch fails context/graph construction instead of allowing the scheduler to move that layer silently.

Before caches are built, Bee also checks whether at least one device serving the tail's layers advertises a native KV-tail attention entry point. If none does, every layer would run the generic tail route, which costs several times more than plain quantized attention, so the context warns and declines the request: --kv-tail-tokens resolves to zero and KVarN keeps its intrinsic 128-token exact suffix. Shared Gemma 4 MTP follows the target context's already-resolved decision. Set LLAMA_KV_TAIL_ALLOW_GENERIC=1 to keep the requested tail on the slower generic route anyway; any value other than an empty string or 0 enables it.

The CLI is parsed once into an immutable, model-independent request. Fit probes and the final context bind that same request to the model's canonical cache groups, so auto, positional, named, and KVarN-minimum policies cannot diverge between estimation and allocation. Public callers that need the same behavior can create a request with llama_kv_tail_request_init, keep it alive through context creation, assign the borrowed pointer to llama_context_params::kv_tail_request, and then free it with llama_kv_tail_request_free.

Let N be the resolved exact length, U the physical ubatch limit, R the advertised suffix-rollback horizon, and S the number of exact streams. Compact persistent exact capacity is (N + R) * S rows and is independent of U. U sizes only graph inputs and reusable transient workspace. Backend buffer alignment may round bytes but does not add logical rows. Exact history remains per logical sequence even with --kv-unified. Positive tails on K-only MLA or DSA attention are rejected during context creation.

R comes from the context's rollback requirement. Contexts that otherwise request no rollback retain one row for the common one-token capability probe; it never defaults to U. The memory capability API reports this bound, and a larger speculative removal must use the checkpoint/reprocess path before cache metadata is mutated.

Partial exact overlays are compatible with --split-mode layer and --split-mode tensor. Layer mode keeps each shadow with its ordinary K/V body. Tensor mode shards standard body/shadow rows, KVarN records and staging, and exact history at complete KV-head boundaries through the model's meta split descriptor. Invalid or unsupported component splits fail during cache construction rather than after graph execution starts.

KVarN's physical staging depth is independent of this logical policy. Increasing -ub may increase transient work but never increases persistent exact coverage. Completed 128-token records are committed eagerly for partial tails, while the canonical exact history stores only N + R rows. A fully covered SWA group uses --kv-tail-type for its W + R compact-native ring and allocates no SWA KVarN records or stage; non-SWA and partially covered groups remain KVarN.

Partial SWA tails retain the upstream-aligned compressed W + U body because older visible rows still use it. Full-window compact-native SWA has no body and stores exactly W + R persistent rows. In both cases current K/V is consumed directly by the same attention softmax before an explicitly ordered history commit.

llama-bench --kv-memory reports cache-owned bytes directly. The kv_k_payload_bytes, kv_v_payload_bytes, kv_exact_history_bytes, kv_rollback_reserve_bytes, kv_staging_bytes, kv_padding_bytes, and kv_resident_bytes fields describe persistent ownership. kv_transient_bytes is the observed reusable CUDA-pool high water and kv_peak_bytes is resident plus transient. The per-route layer counters show native bodyless, native mixed, planned device-fallback, and CPU layers. These fields are more precise than deriving cache memory from whole-process VRAM; the separate CUDA/WDDM fields remain useful for reconciliation and spill detection.

Overlay state uses a framed standard-memory section. Exact restore requires the same structural group, resolved length, representation, and exact type. Native exact state is carried by the ordinary body and does not serialize a duplicate shadow. The extended full and sequence state APIs accept LLAMA_STATE_SEQ_FLAGS_BODY_ONLY to deliberately omit overlay shadows; loading that state into a tail-enabled context is valid, but the coverage API reports LLAMA_KV_TAIL_DEGRADED_BODY_ONLY_STATE until new writes refill the recent window. Server metrics expose requested/exact token totals, complete/partial/no coverage group counts, and degraded-sequence counts.

KVarN state version 15 stores sequence-selective logical compressed record groups, compact exact payloads, selected stage rows, and destination remapping independently of ubatch workspace, so state may move between ub=128 and ub=512. Live checkpoint state may retain sealed history already owned by the context; LLAMA_STATE_SEQ_FLAGS_SELF_CONTAINED exports every payload needed after the source is removed. Compatible version 12 and 13 state remains readable; version 11 is rejected rather than reinterpreting its old physical workspace layout. Tail length, type, preset, rollback horizon, representation, and structural-group mismatches fail closed.

Sequence state writes precision-tail manifest version 5, including exact source cell/generation identities, exact-tail local slots, insertion order, the per-sequence write cursor, the compact representation, and rollback horizon, and supports host or on-device tensor transfer. Version 4 remains readable; manifest version 2 remains readable for non-compact layouts, while version 1 restores conservative degraded provenance. Immediate body membership and position changes after sequence copy are preserved; pending exact rows materialize as one batch when state data is requested.

Restore publishes no tensor or metadata changes until the complete state frame has validated. A truncated, corrupt, mismatched, or failed backend transfer is cancelled. Deferred precision-tail copy failures propagate through immediate state save and subsequent decode instead of being reported as successful.

Prompt-cache message boundaries do not reset the suffix. Standard and KVarN state is sequence-selective in unified and non-unified layouts. The planner records lexical, restorable, and committed token counts and restores target, draft, and speculative state as one prepared transaction. Live slots and RAM entries are compared by safe restorable tokens before existing tie-breaks. Standard and recurrent caches keep upstream prompt batching; KVarN alone adds its descriptor-boundary eligibility rule. RAM entries use self-contained immutable state, are repeatably restorable, and clear an idle unified slot only after successful admission. Tail length affects quality, memory, and transfer cost, not logical prompt-cache eligibility.

KVarN durable prompt checkpoints remain on complete 128-token descriptor boundaries for the current G128 presets. The transient live exact frontier may still service the existing bounded speculative micro-rollback contract; it does not turn sealed history into arbitrary-position records. Standard KV has no KVarN group constraint and may restore any validated logical position. For standard caches with a precision tail, a hot partial checkpoint references the still-live body and transfers its logical manifest and exact overlay. The manifest remains proportional to the logical prefix, so this is not a native sealed-block arena or a strictly frontier-only operation. Copy-on-write sharing applies to serialized checkpoint byte buffers, not native KV blocks. Self-contained RAM state continues to own and transfer the body because it must survive source removal.

Completion timing JSON includes cache_lcp_n, cache_planned_n, cache_reprocessed_n, cache_source, and cache_reason. Prometheus exports prompt_cache_admission_*_total, prompt_cache_restore_*_total, prompt_cache_accounted_bytes, n_busy_slots_per_decode, and the kv_tail_* coverage/degradation gauges. A nonzero restore-failure or degraded tail metric is actionable rather than silently counted as a hit. Accounted bytes are serialized payload accounting, not exact process-resident memory.

The first five rows are upstream speculative controls with Bee-specific DFlash behavior. The --spec-dm-* rows are Bee server additions.

GET /models lists model identity, status, source, aliases, tags, and capabilities. Matching upstream, each entry's status exposes the child argv (status.args) and, for preset-backed models, the resolved INI preset (status.preset) with sensitive options stripped; these may contain local paths for custom-path preset models. It ignores former reload query parameters. Refresh model sources with POST /models/reload; when --api-key is configured this mutation requires the same Authorization: Bearer ... or X-Api-Key authentication as other non-public routes. --hf-token is a sensitive option: router children receive it through HF_TOKEN, never through argv or serialized presets.

See INI presets for syntax, inheritance, remote presets, and a Bee configuration example.

Use the same corpus, context, logical batch, and physical ubatch for both KLD legs.

Release packages are built with CUDA 12.4 and 13.3. CUDA 12.4 can emit the Maxwell, Pascal, and Volta PTX targets used by the portable KVarN route; CUDA 13.3 covers Turing and newer architectures. The release workflow no longer has an exhaustive per-architecture CUDA compile gate. For a local or CI build, select the intended target explicitly with CMAKE_CUDA_ARCHITECTURES when the build host cannot detect it. Pre-Turing support remains runtime-unqualified until matching real devices pass the KVarN parity, memory, and model-smoke tests.