mxfp4

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against the official code.

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-compatible server with tool calling + hybrid cloud relay; CPU, Apple Silicon & CUDA (MLX).

Run the official Kimi K3 MoE checkpoint on one 128 GB AMD Strix Halo box. ROCm-resident static weights, MXFP4 experts streamed from NVMe via io_uring. C engine, chat client, OpenAI-compatible server. Very experimental.

Run Kimi-K3 (2.8T MoE) from SSD with a tiny readable C++17 engine—native MXFP4, io_uring, and O_DIRECT.

CPU inference for Kimi K3, a 2.78T-parameter MoE LLM, in pure Rust. No GPU, no BLAS, no PyTorch. Streams the checkpoint from disk. Byte-identical port of kimi-k3-in-c.

面向 8×RTX 4090（SM89）的 DeepSeek-V4-Flash-0731 MXFP4 SGLang 实验分支

Qwen3-8B quantization study across vLLM, TensorRT-LLM, AutoRound, INT8, and MXFP4

Bit-exact conformance and verification suite for OCP Microscaling (MX) arithmetic in RTL. Operation-layer golden reference for MXFP8/MXFP4/NVFP4.

Serving DeepSeek-V4-Flash-0731 at 1M context across 2x DGX Spark (GB10) with TP=2, DSpark speculative decoding and NVFP4 KV cache. Measured findings: why PP is impossible, DSpark's 32-stream ceiling, speculation 1.4-1.9x, MXFP4-vs-NVFP4.

Run Kimi K3 and DeepSeek-V4-Flash-0731 MoE architectures with this zero-dependency Rust engine verified against official reference code.

A Codex skill for designing, implementing, and evaluating post-training quantization research with block-scaled formats.

GRIT — Grouped Reduced-precision Interchange Type: a checkable 64-byte descriptor + contract for block-scaled, sparse, sub-byte tensors. Spec, conformance vectors, and 5 zero-dependency implementations (C, C++, Rust, Python, TypeScript). grit scan audits GGUF/safetensors files you already have.

Field notes from optimizing LLM inference on an NVIDIA DGX Spark. Everything measured on the machine.

Deploy DeepSeek-V4-Flash-0731 on dual NVIDIA DGX Spark clusters with vLLM, achieving 60–70 tok/s and 1M context via 200GbE direct connection.