The README pointed core primitives (Preprocessor, TorchDataset, metrics,
MSLoader, SyntheticDataGenerator) at samrfi.* paths that no longer export
them; these live in the external rfi_toolbox package, verified against its
__all__ lists. Repoint all such imports to rfi_toolbox.

Also fix the Python API example bodies to match real signatures:
- SyntheticDataGenerator(config).generate(output_path=...) via ConfigLoader,
  not the nonexistent config_path/num_samples/output_dir kwargs
- Training example uses BatchedDataset + the .dataset wrapper SAM2Trainer
  actually expects, dropping the nonexistent TorchDataset.from_directory and
  output_dir/save_best_only kwargs

Co-Authored-By: Claude Opus 4.8 <[email protected]>

1. learning_rate default 1e-6 -> 1e-5 to match docstring and configs.
2. Best-model checkpoint falls back to training loss when no validation
   set is given, instead of saving nothing.
3. Opt-in early stopping via `patience` kwarg (default None = unchanged),
   threaded through TrainingConfig, the YAML flatten path, and the CLI.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

samrfi carried local forks of rfi_toolbox's shared data layer from a
half-finished migration. The forks were stale (notably ms_loader.py lacked
the field_id support rfi_toolbox has) and several package-level imports of
them were already broken (samrfi/__init__ __all__ listed names it never
imported; `from samrfi.data import MSLoader` and `from samrfi.evaluation
import ...` raised because those __init__ files export nothing).

Make rfi_toolbox the single source of truth for shared primitives:

- Repoint all consumers (cli, adaptive_patcher, ms_explorer, ms_generator,
  hf_dataset_wrapper, tests, validation scripts, docs) to rfi_toolbox for
  MSLoader, Preprocessor, TorchDataset, BatchWriter, SyntheticDataGenerator,
  and the evaluation metrics. cli/adaptive_patcher now get field_id support.
- Delete the orphaned forks: data/{ms_loader,preprocessor,torch_dataset}.py,
  data_generation/synthetic_generator.py, and
  evaluation/{metrics,statistics,ms_injection}.py.
- Fix samrfi/__init__ __all__ to list only what samrfi actually exports
  (SAM2-specific classes + ConfigLoader) and update module docstrings to
  state that shared primitives live in rfi_toolbox.

SAM2-specific classes (SAMDataset, BatchedDataset, RAMCachedDataset,
HFDatasetWrapper, MSDataGenerator) are unchanged. Verified by inspection
that rfi_toolbox's versions cover every method used at the call sites; the
full test suite must be run in the GPU/CASA environment to confirm.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The evaluation/ package had no remaining content after the rfi_toolbox
migration (all metrics live in rfi_toolbox.evaluation) and nothing imported
it, so drop the empty package. Also correct the module-level usage example
to wrap the dataset in the `.dataset`-bearing object SAM2Trainer requires.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

Add two opt-in throughput/memory levers to SAM2Trainer.train(), both
defaulting to off so existing fp32 runs are unchanged:

- use_amp: wraps the train and validation forward passes in autocast and
  scales the loss with a GradScaler. CUDA-only; silently no-ops on CPU.
  Uses the torch>=2.3 device API with a fallback to torch.cuda.amp for
  2.0-2.2.
- accumulation_steps: normalizes the loss by the step count and only calls
  optimizer.step()/zero_grad() every N batches (and on the final batch to
  flush the remainder), giving an effective batch size of
  batch_size * accumulation_steps without the memory cost.

Both are threaded through TrainingConfig (with validation), the YAML flatten
path, and the CLI. Reported per-batch loss is rescaled back to its
un-normalized magnitude so logged values stay comparable across settings.

Needs GPU validation: confirm AMP numerics (no NaNs/scaler stalls) and that
accumulation reproduces large-batch behavior, in the deployment environment.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

Two opt-in convergence levers for SAM2Trainer.train(), defaulting off:

- scheduler ('cosine' | 'linear' | None) with warmup_steps: builds a
  transformers warmup schedule over the total optimizer-step count and steps
  it once per optimizer step (accounting for accumulation). Scheduler state
  is saved in both checkpoint paths and restored on resume.
- encoder_lr: when the vision encoder is unfrozen (freeze_vision_encoder=
  False), gives it a separate, typically lower LR via optimizer parameter
  groups. None collapses to a single LR (no behavior change).

Threaded through TrainingConfig (with validation), the YAML flatten path,
and the CLI. Defaults preserve the existing constant-LR, single-group setup.

Needs GPU validation alongside Phase 2: confirm the schedule shape, resume
restores the LR correctly, and encoder fine-tuning is stable vs the frozen
baseline before adopting.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

Lightweight alternative to the SAM2 path: frozen DINOv3/v2 ViT encoder
(21.6M) + DPT-style decoder (3.17M trainable) producing full-resolution
masks, no prompts. amplitude / realimag complex input modes.

- training/dino_segmenter.py: backbone-agnostic frozen ViT + DPT decoder
- training/dino_trainer.py: DiceCE loss, metrics, fit loop, complex dataset
- evaluation/sumthreshold.py: faithful Offringa-2010 SumThreshold + MAD
- evaluation/buried_metrics.py: recall-vs-local-SNR, dice/precision/recall
- scripts/: config-driven sample inspection, overfit probe, buried test,
  DINO-vs-classical comparison
- configs/: inspect_single, dino_overfit
- pixi.toml/lock: CPU dev env

Held-out buried-RFI result (RFISimulator, amplitude mode, DINOv2-small):
DINO recovers 75-82% of sub-noise (<1 sigma) RFI vs SumThreshold 4-12%,
MAD 0%, at 0.96 precision. calcquality (v1 metric) is blind to this.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

scripts/dino_predict_ms.py: load a real MS baseline via rfi_toolbox MSLoader,
run the (synthetic-trained) DINO segmenter, and overlay SumThreshold and
optional tfcrop+rflag masks for comparison. Caches the trained decoder.

pixi: add casatools + casatasks for MS read and flagdata.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

Wire the DINO frozen-backbone segmenter to the coherent-phase simulator and add
a proper training scaffold (kills the 12-patch overfit caveat).

- sim_data.SimBatchStream: on-device full-plane generation via RFISimulatorTorch
  (no shards/workers, GPU stays hot); fixed-seed make_val_set for stable metrics.
- dino_trainer.train: validation loop, buried/bright recall split at per-sample
  noise p99 (the thesis metric), best-checkpoint by buried recall (decoder-only
  + metadata), cosine schedule, early stopping.
- scripts/dino_train.py: entry point; realimag primary + amplitude A/B, --smoke.
- pixi: CUDA pytorch build (cuda=12 system req + build pin), pytest.

Validated end-to-end on a local GPU (1080 Ti); fits 1024^2 bs4 at ~6 GB.

Co-Authored-By: Claude Opus 4.8 <[email protected]>