A comprehensive toolkit for training and running lightweight adapters for GGUF-based language models (ERNIE, Llama, Mistral, Phi-3, etc.) without modifying the base model.
Now with two adapter architectures (external and universal) and automatic type detection.

• Overview
• Installation
• Training (train-adapter)
  • Quick Start
  • Detailed Options
  • Examples
• Inference (run-inference)
  • Quick Start
  • Operation Modes
  • Detailed Options
  • Examples
• Methodology
  • External Adapter
  • Universal Adapter
  • Cost-Efficient Training
  • Transfer Learning
• Results & Performance
• Troubleshooting
• License

This toolkit implements External Logit Correction, a novel approach for domain adaptation of quantized LLMs. Instead of fine‑tuning the entire model (which is impossible with GGUF), we train a lightweight external adapter that refines the base model's logits. The adapter is:

• Lightweight: Typically 256‑512 dimensions vs. billions in base model
• Fast to train: Hours on consumer GPU vs. days for full fine‑tuning
• Transferable: Adapters trained on small models work on larger family members
• Non‑invasive: Base model remains completely unchanged

The system now supports two adapter architectures:

• External Adapter (original): full‑vocabulary correction, ideal for models where you can afford larger adapter dimensions.
• Universal Adapter (cross‑model): top‑k correction using semantic embeddings, designed to be more transferable across model sizes and quantizations.

Both architectures are auto‑detected during loading, so you never need to specify which one you are using – the toolkit reads the configuration or inspects the state dict.

git clone https://github.com/ShotokanOSS/ggufForge.git
cd ggufForge

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Standard Installation:

pip install -U pip
pip install .

For CUDA Support (GPU Acceleration):

CMAKE_ARGS="-DLLAMA_CUDA=on" pip install .

For Development with Additional Tools:

pip install .[dev]

train-adapter --help
run-inference --help

Train lightweight adapters for GGUF models using streaming datasets.
The training script can automatically continue from an existing adapter repository – just pass its Hugging Face ID as --model.

Basic training (default: universal adapter) on ERNIE model:

train-adapter --model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF

Train external adapter on Llama 2 with custom parameters:

train-adapter \
  --model TheBloke/Llama-2-7B-GGUF \
  --filename llama-2-7b.Q4_K_M.gguf \
  --adapter-type external \
  --adapter-dim 512 \
  --steps 5000 \
  --learning-rate 3e-5

Continue training an existing adapter from Hugging Face:

train-adapter --model my-username/my-adapter-repo --resume

1. Fast universal adapter training on dense model (Phi‑3, 1000 steps):

train-adapter \
  --model microsoft/Phi-3.5-mini-instruct-GGUF \
  --steps 1000 \
  --learning-rate 5e-5 \
  --adapter-dim 256 \
  --adapter-type universal

2. Extended external adapter training on ultra‑low‑bit MoE (ERNIE, 14000 steps):

train-adapter \
  --model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
  --filename ERNIE-4.5-21B-A3B-Thinking-UD-Q2_K_XL.gguf \
  --adapter-type external \
  --steps 14000 \
  --adapter-dim 512 \
  --output-dir ernie-external-adapter

3. Custom dataset training with periodic evaluation:

train-adapter \
  --model TheBloke/Mistral-7B-Instruct-v0.1-GGUF \
  --dataset my-org/custom-dataset \
  --prompt-col instruction \
  --output-col response \
  --eval-steps 500 \
  --log-file my_log.csv \
  --hf-repo my-org/mistral-universal-adapter

4. Evaluation only:

train-adapter \
  --model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF \
  --eval-only \
  --checkpoint checkpoints/adapter_final.pt

5. Resume interrupted training:

train-adapter \
  --model my-org/llama-adapter \
  --resume \
  --steps 20000  # new total steps

Run inference with or without adapters, compare models, or chat interactively.
The inference script automatically detects the adapter type (external or universal) from the loaded weights – no extra flags needed.

Single question with adapter (auto‑detects type):

run-inference \
  --mode single \
  --question "What is machine learning?" \
  --adapter true

Interactive chat:

run-inference --mode chat

Compare base vs. adapter:

run-inference \
  --mode compare \
  --question "Explain quantum computing"

Note: For universal adapters, the top‑k value is fixed at 50 (as used during training). This is not configurable via CLI.

1. Single question with custom parameters (adapter auto‑detected):

run-inference \
  --mode single \
  --question "Explain the theory of relativity" \
  --temperature 0.7 \
  --max-tokens 1000 \
  --min-p 0.1 \
  --output response.txt

2. Chat with custom system prompt (adapter disabled):

run-inference \
  --mode chat \
  --system-prompt "You are a helpful physics tutor. Explain concepts simply." \
  --temperature 0.5 \
  --adapter false

3. Compare with file input:

run-inference \
  --mode compare \
  --file question.txt \
  --max-tokens 500 \
  --output comparison.json

4. Interactive menu mode (full control):

run-inference --mode interactive

5. Custom model configuration (different base + adapter):

run-inference \
  --mode single \
  --base-repo TheBloke/Llama-2-7B-GGUF \
  --gguf-filename llama-2-7b.Q4_K_M.gguf \
  --adapter-repo my-org/llama-universal-adapter \
  --context-size 4096 \
  --question "Tell me a story"

The original architecture – a single‑block causal transformer that receives token IDs and full base logits as input, and outputs correction logits for the entire vocabulary.

Input: [token_ids, base_logits] → Token Embedding + Logit Compressor → Additive Fusion → LayerNorm → 
8‑Head Causal Attention → Feed‑Forward Network (4× expansion) → Output Head → Corrected Logits

• Vocabulary‑aware: operates on the full logit vector, allowing fine‑grained corrections.
• Cache‑compatible: can reuse KV‑cache for efficient generation.
• Best for: models where you can afford the extra memory of full‑vocabulary correction.

A more transferable architecture that only corrects the top‑k tokens of the base logits, using semantic embeddings of the token candidates. This makes the adapter less dependent on the exact vocabulary and more robust to model size changes.

Base logits → top‑k (tokens + logits) → token IDs → semantic embeddings (via Sentence‑Transformer) →
concatenate with logits → projection → Multi‑head Self‑Attention → FFN → correction scores for top‑k tokens

The final logits become:

final_logits = base_logits + scatter_add(top_k_corrections)

• Cross‑model transfer: adapters trained on small models work on larger family members.
• Memory efficient: only processes top‑k tokens (e.g., 50) instead of the full vocabulary.
• Best for: large vocabularies, model families, and ultra‑low‑bit quantizations.

Both architectures are auto‑detected during loading: the toolkit inspects the state dict keys or the saved config.json and selects the correct inference path automatically.

Why it's efficient:

1. Tiny parameter count: ~1M vs. billions in base model
2. Short training: Hours instead of days
3. Transfer learning: Train small → use on large
4. Streaming data: No dataset download needed

Universal adapters show remarkable transfer capabilities:

Requirements for transfer:

1. Same model family
2. Compatible vocabulary
3. Similar quantization scheme

1. Greater degradation, greater improvement: Ultra‑low‑bit models benefit most.
2. Rapid convergence: Dense models need only 1,000 steps.
3. Universal adapter transfers: Improvements hold across model sizes.
4. Consistent gains: Improvements are robust across validation sets.

1. Model loading fails:

# Check available files
run-inference --base-repo TheBloke/Llama-2-7B-GGUF --verbose
# Specify exact filename
run-inference --base-repo TheBloke/Llama-2-7B-GGUF --gguf-filename llama-2-7b.Q4_K_M.gguf

2. Out of memory:

# Reduce context size
run-inference --context-size 2048
# Use CPU layers
run-inference --gpu-layers 20
# For training, lower accumulation steps or adapter dimension
train-adapter --adapter-dim 128 --accumulation-steps 16

3. Slow generation:

# Reduce adapter window (external adapter only)
run-inference --adapter-window 1024
# Use base model only
run-inference --adapter false
# Disable summaries
run-inference --summary false

4. Poor quality responses:

# Adjust temperature
run-inference --temperature 0.3  # More focused
run-inference --temperature 0.9  # More creative
# Adjust min‑P
run-inference --min-p 0.01  # More diverse
run-inference --min-p 0.2   # More conservative

5. Adapter type not recognized: The toolkit auto‑detects the type from the state dict. If it fails, ensure the adapter was saved with the correct config.json (including adapter_type field). You can also manually inspect:

python -c "import torch; sd = torch.load('adapter_final.pt', map_location='cpu'); print(sd.keys())"

Enable verbose mode:

run-inference --verbose --mode single --question "Test"

Check model configuration:

train-adapter --model unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF --eval-only --verbose

Monitor GPU usage:

nvidia-smi -l 1  # Linux
# or use --no-progress to reduce overhead during inference
run-inference --no-progress --mode chat

This project is licensed under the Apache License 2.0. See LICENSE for details.

Key Points:

• Free for commercial and research use
• Attribution required
• No warranty provided
• Patent rights granted

For research paper, detailed methodology, and extended results, see Study Note: The transfer just works stable inside of one Model family yet-we are Still working on transfer between different model familys