Strike

Strike is a lightweight Go sidecar agent that gives teams running self-hosted LLM inference real-time visibility into what each request costs and how their GPUs are being used.

Teams serving models on their own hardware (vLLM, TGI, a FastAPI wrapper, …) can see GPU dashboards and request logs, but usually can't answer the questions that matter to the business:

• How much did that request / customer / model cost us to serve?
• Which tenant is driving our GPU spend?
• Are our GPUs saturated, or are we paying for idle silicon?
• What is our cost-per-1K-tokens in practice, not in theory?

Strike sits next to the inference server, observes every request, continuously samples the GPU, attributes a dollar cost to each unit of work, and exports it as metrics.

Full design rationale lives in DESIGN.md.

Client request → Inference server (vLLM / TGI / FastAPI)
                       │  hooks / middleware report each request
                       ▼
                 POST /v1/events
                       │
                 Strike agent
                   ├─ events.Processor  → fans each event out to consumers
                   │     ├─ cost engine     (prices tokens, aggregates by model/tenant)
                   │     └─ inference tracker (logging / forwarding)
                   └─ GPU collector     → samples nvidia-smi / NVML on a timer
                       │
                       ▼
                 Prometheus /metrics, logs

1. Tracks inference requests — each request is reported as an InferenceEvent (request ID, model, input/output token counts, latency, timestamp, optional tenant ID).
2. Samples GPU utilization — a background collector reads GPU state (utilization, memory, temperature, power) on a timer via NVML (the NVIDIA Management Library, through the go-nvml bindings). Multiple sampler backends can be registered; a host without an NVIDIA driver simply reports no backend.
3. Prices each request — the cost engine applies a per-model pricing table (USD per 1K input/output tokens) and keeps running totals by model and tenant.
4. Exposes metrics — costs and GPU stats are published for Prometheus and the bundled Grafana dashboard.

• Go 1.23+.
• CGO_ENABLED=1 and a C compiler (gcc/clang) — the NVML bindings do not compile with cgo disabled. (The provided Docker build handles this for you.)
• curl (for the load-test script).
• For real GPU samples: an NVIDIA driver/libnvidia-ml.so available to the process. In a container this is provided by the NVIDIA container runtime (NVIDIA_VISIBLE_DEVICES + NVIDIA_DRIVER_CAPABILITIES=utility). On a Mac or CPU-only host the GPU sampler reports no backend available and the rest of the agent runs fine.

Run the agent and drive synthetic traffic at it:

# Terminal 1 — build and run the agent (serves on :5231)
./scripts/run-local.sh

# Terminal 2 — fire randomized inference events at it
./scripts/simulate-load.sh

You should see the agent log each event being received, tracked, and priced, e.g.:

level=INFO msg="priced inference event" request_id=req-1 model=gpt-4o tokens_in=1000 tokens_out=500 cost_usd=0.0125

curl -s -X POST http://localhost:5231/v1/events \
  -H 'Content-Type: application/json' \
  -d '{"id":"req-1","model":"gpt-4o","tokens_in":1000,"tokens_out":500,"latency_ms":850,"tenant_id":"acme"}'

The agent serves on :5231 by default (override with STRIKE_SERVER_ADDR).

{
  "id": "req-1",
  "model": "gpt-4o",
  "tokens_in": 1000,
  "tokens_out": 500,
  "latency_ms": 850,
  "tenant_id": "acme"
}

id, model, tokens_in, tokens_out, and latency_ms are required. tenant_id is optional; events without it are attributed to tenant unknown.

All settings are read from the environment at startup (via cleanenv), with the defaults below. Invalid values cause the agent to exit; non-positive buffer or interval values fall back to the default.

Durations use Go syntax (250ms, 2s, 1m).

Per-model rates (USD per 1K input/output tokens) live in internal/cost/pricing.go. Unknown models fall back to a non-zero default so they are still attributed some cost rather than reporting $0. Cost for an event is:

(tokens_in / 1000) * input_rate + (tokens_out / 1000) * output_rate

Prometheus metrics are exposed at GET /metrics. Inference metrics are labelled by tenant_id and model; GPU metrics by gpu_uuid/gpu_name.

• scripts/run-local.sh — build/run the agent (--build compiles to ./bin/strike-agent first).
• scripts/simulate-load.sh — POST randomized events. Configurable via env: STRIKE_URL, REQUESTS (0 = infinite), CONCURRENCY, DELAY.

The build is a multi-stage Dockerfile (cgo build on golang:bookworm, runtime on debian:bookworm-slim, runs as a non-root user). The NVIDIA driver is not bundled — the container runtime injects it at start.

# Build (context must be the repo root)
docker build -f deployments/docker/Dockerfile -t strike-agent .
docker run -p 5231:5231 strike-agent

# Or via compose
cd deployments/docker && docker compose up --build

To give the container GPU access, uncomment the GPU block in docker-compose.yml (requires the nvidia-container-toolkit on the host).

Manifests in deployments/k8s/ cover the two deployment modes from DESIGN.md:

• sidecar.yaml — a native-sidecar pod-spec fragment injected into an inference workload's pod, for clean per-workload cost/GPU attribution.
• daemonset.yaml — one agent per node for node-level monitoring of all GPUs on the host.

Both grant GPU visibility with NVIDIA_VISIBLE_DEVICES + NVIDIA_DRIVER_CAPABILITIES=utility (monitor-only, no CUDA) and read tunables from pod annotations via the downward API. A monitoring agent intentionally does not request the nvidia.com/gpu resource, so it never reserves a GPU from real workloads.

cmd/agent/          main — loads config, wires the agent, handles shutdown
internal/
  api/              HTTP server, routes, handlers
  collector/        inference + GPU collectors
  cost/             cost engine + per-model pricing table
  events/           event processor (fan-out to consumers)
  gpu/              GPU sampler backends (NVML implemented; nvidia-smi stub)
  logger/           shared slog logger
  config/           env-based configuration (cleanenv)
  metrics/          Prometheus definitions + registry
pkg/schema/         shared InferenceEvent / GpuSample types (wire contracts)
deployments/        Docker + k8s (sidecar / daemonset) manifests
dashboards/grafana/ Grafana dashboard
scripts/            run-local.sh, simulate-load.sh

See DESIGN.md for the architecture, the GPU-visibility model (how the sidecar actually sees the GPU), and the event-processing layer.

Working end to end: request ingestion → fan-out → cost pricing (per model and tenant) → Prometheus metrics, plus NVML GPU sampling on a timer, env-based config, graceful shutdown, and Docker/Kubernetes packaging.

Still rough / planned: the nvidia-smi sampler backend (NVML is the implemented one), broader pricing coverage, and richer /v1/stats.