Detect and quarantine wedged vGPU VFs - #435

Detection moves from host-side matching of the raw NVRM kernel line to an
explicit guest-to-host report: the guest agent watches /dev/kmsg for the
driver's RmInitAdapter failure and emits a HYPEMAN-GPU-INIT-FAILED marker
over the established sentinel-marker channel, which the controller convicts
on. The raw kernel line is no longer matched, so detection now requires the
guest agent; images without it do not report.

Quarantine hardening in the same pass:

- The conviction brake pauses instead of drops: a suppressed conviction
  leaves the tail open and the agent's re-emission retries it once the
  window clears.
- A match on an already-quarantined VF is not a new conviction: no brake
  accounting, no metric, no wedge-count inflation on controller restarts.
- Tails reset when the instance acquires a new assignment, so a finished
  tail from a previous boot or VF cannot suppress scanning the next one.
- A vf-health state file that fails to load refuses mutations (and retries
  the load) instead of letting the next conviction clobber every previously
  persisted quarantine.
- Oversized unterminated log lines are skipped instead of re-buffered on
  every scan.
- The controller idles on hosts without the vendor VFIO framework.

GPU.md: DCGM quiesce is now an ordered step of the recovery sequence, and
clearing vf-health.json documents the immediate-restart requirement.

Review fixes on the quarantine layer:

- start now archives the previous boot's serial log before persisting the
  new vGPU assignment. The sentinel keys its tail on the assignment epoch,
  so the old ordering let a scan replay the previous boot's wedge report
  against the freshly assigned VF. A failed archive is fatal for GPU
  instances instead of a warning.
- placement and profile availability refuse to run when the VF health
  state file exists but cannot be loaded, instead of treating the empty
  in-memory set as healthy and returning every quarantined VF to
  rotation. The load is retried on each attempt, so a repaired file
  self-heals.
- a conviction whose persist fails is rolled back from memory: keeping it
  made the next report look like a repeat conviction, ending retries with
  nothing on disk.
- the marker pattern requires the full shape through the quoted NVRM
  payload; a truncated or payload-less marker echoed by an exec command
  no longer matches.
- unreadable instance metadata in the sentinel target listing logs a
  warning instead of silently shrinking detection coverage.
- GPU.md recovery runbook unwound a circularity: placement excludes
  quarantined VFs and there is no VF-pin API, so the entry is cleared
  before the verification boot; the sentinel re-quarantines automatically
  if the cycle did not cure the VF.

The /dev/kmsg record priority encodes facility*8+level. Kernel printk is
always facility 0 and the kernel assigns userspace writers LOG_USER or
higher (a facility-0 prefix is coerced to LOG_USER, verified on a live
6.12 kernel), so requiring facility 0 makes in-guest forgery of the
report impossible, matching the intended kernel-records-only semantics.

Also retry a failed /dev/kmsg open instead of permanently disabling the
watcher for the guest's lifetime.

The scan buffered each complete line whole, so guest console output
could make the controller allocate line-sized buffers every pass; the
64KB cap only kept an unterminated tail from being re-read, and once
such a line was skipped its late-arriving tail was parsed as a fresh
line. Read through a fixed-size buffer instead: a line that overflows
it cannot be a marker, so it is discarded — across scans if its newline
has not arrived — without ever being held in memory, and its tail can
no longer replay a marker. Rotation resets the skip state with the
offset.

When the persisted state file fails to load, quarantine mutations are
refused and vGPU placement fails closed, but the quarantined-VFs gauge
reads zero from the empty in-memory set — exactly when quarantines
exist and are unreadable. Export the load-failure state as its own
gauge so the condition is alertable.

The serial console is a shared byte stream: kernel printk bypasses the
tty buffer and can land mid-marker, and on a wedged VF the kernel is
emitting NVRM errors exactly when the agent reports. A corrupted copy
does not match the host's full-shape scan (deliberately, so echoed
commands cannot convict), which delayed the report to the next 30s
re-emission. Emit each report as three identical lines sharing one ts;
the host convicts on the first intact copy and ignores the rest.

Auto-conviction no longer pauses on a burst. Systemic non-wedge init
failures (e.g. a driver-mismatch image rollout) are expected to be
caught on a test host before reaching production, and the convictions
counter remains the alerting signal if one gets through; the brake was
extra state and logic guarding against a case the rollout process
already covers. Quarantine still only removes capacity and never
touches instances, and the store's fail-closed load handling is
unchanged.

Each /dev/kmsg read returns exactly one record and fails with EINVAL —
without consuming the record — when the buffer is smaller. Records run
up to CONSOLE_EXT_LOG_MAX (8 KiB), so bufio's default 4 KiB buffer
wedged the watcher on the first oversized record: every reopen replayed
the ring into the same record, silently losing all detection behind it.
Size the buffer to the kernel's record bound and log non-EPIPE scan
errors so a wedge is visible instead of silent.

A quarantine is only real once it is on disk, but the persist renamed
without syncing the file or directory, so a host crash right after a
conviction could silently drop it. Sync the temp file before the rename
and the directory after, and deduplicate the sorted record listing.

The matched line is guest-controlled console bytes up to the 64 KiB
line cap; logging and persisting it verbatim put up to that much guest
output in error logs and vf-health.json. Keep just the marker match.

ClearVFQuarantine had no callers — the runbook documents editing
vf-health.json directly — and WedgeCount was always 1. QuarantineVF's
record return and IsVFQuarantined only served tests, and the sentinel's
isQuarantined pre-check duplicated what QuarantineVF already reports
via existed. The hostFramework indirection wrapped a single
DiscoverVGPU call nothing injected.

A transient DiscoverVGPU error at startup failed open permanently,
leaving a host that is not vendor VFIO scanning instance metadata every
tick for the process lifetime. Keep failing open while discovery
errors, but retry the probe each tick so the controller exits once the
host is known not to be vendor VFIO.

A conviction used the metadata snapshot taken before the log read, so a
stop/start racing the scan could quarantine the VF the instance no
longer holds. Reload the assignment and convict only when the VF and
assignment epoch are unchanged; start's archive-before-persist ordering
guarantees a marker read under an unchanged epoch belongs to that
epoch's VF. On any other outcome the tail stays open and the next scan
picks up the current assignment.

Target listing used the lenient metadata walk, which drops entries
whose stat fails without ever reaching the unreadable-metadata warning.
Walk strictly first so a stat failure names the instance losing
detection coverage, then fall back to the lenient walk so the readable
rest keeps being scanned.

The remediation section required an empty card but never said how to
get one: the quarantine only deprioritizes the card, so under capacity
pressure new placements keep refilling its healthy VFs. Document the
manual cordon (pre-quarantining the card's VFs in vf-health.json plus a
hypeman restart), why running instances are untouched and drain through
their normal lifecycle, and how to monitor and uncordon.

The pre-conviction recheck treated a missing assignment like a changed
one and skipped the quarantine. A stop or delete landing between the
metadata snapshot and the recheck releases the assignment but cannot
un-wedge the VF, and the archive-before-persist ordering already ties a
marker read under this epoch's snapshot to this epoch's VF, so the
conviction is valid. Skip only when a different assignment exists.