rtx-3090

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

THE FIRST DLSS 5 MANAGER

Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): TurboQuant KV cache, MTP speculative decoding with a 64K draft-vocabulary shortlist, custom SM86 + Qwen kernels. 90 tok/s over a 100K-token generation at temperature 1.

Qwen3.8-Flash-Next on 2× RTX 3090 + 128 GB RAM. Experimental peaks: 1,860 tok/s prefill at 260K input; 89.1 tok/s decode at 128K input. Full 256K context.

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from the committed data by a checker that fails on drift, a coverage probe reports how many of them it actually covers, and ERRATA.md lists this study's own retracted claims.

Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.

The bare metal in my basement

Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.

Qwen3.8 Flash Next AWQ INT4 4x3090 v1: 262K, Vision, FP8 QSA KV and CUDA Graphs on 4x RTX 3090

Qwen3.6 vLLM Toolkit — Launcher + Templates, Optimized for 2×24 GB. vLLM launcher with an optimized hybrid chat template drawing from the best community fixes for Qwen 3.6 27B.

Qwen3.8-Flash-Next on two RTX 3090s: what it takes, how fast it goes, and what could not be measured

DeepSeek-V4-Flash-0731 on 8x RTX 3090 (SM86) with vLLM — verified FP8 serving, benchmarks, build guide, and reproducible release.

Local agentic coding stack: Hermes Agent + Qwen3.5-27B + GLM-4.7-Flash on dual RTX 3090s. Daily-driver agentic work, no cloud, no metering. Companion to blog.zacharycangemi.com.

Qwen3.8-Flash-Next on 4x RTX 3090 with vLLM: 806,792-token KV pool, three 262K sessions resident, MTP, host-mapped PLE, pinned build and container recipes.

Hand-written CUDA vs Mojo GPU kernels benchmarked on consumer Ampere (RTX 3090, sm_86), with roofline analysis

Dual RTX 3090 local LLM infrastructure — 48 GB VRAM AI cluster build. Hardware spec, photos, power-limit script. Companion to the blog series at blog.zacharycangemi.com.

Prebuilt Linux sm_86 binaries for the NInfer RTX 3090 engine (upstream ships Windows only) + a bash CLI that rents the 3090 on Vast.ai and pays $0.00 when idle.