labml.ai Deep Learning Paper Implementations

This is a collection of simple PyTorch implementations of neural networks and related algorithms. These implementations are documented with explanations,

The website renders these as side-by-side formatted notes. We believe these would help you understand these algorithms better.

We are actively maintaining this repo and adding new implementations almost weekly. for updates.

• Multi-headed attention
• Transformer building blocks
• Transformer XL
  • Relative multi-headed attention
• Rotary Positional Embeddings
• Attention with Linear Biases (ALiBi)
• RETRO
• Compressive Transformer
• GPT Architecture
• GLU Variants
• kNN-LM: Generalization through Memorization
• Feedback Transformer
• Switch Transformer
• Fast Weights Transformer
• FNet
• Attention Free Transformer
• Masked Language Model
• MLP-Mixer: An all-MLP Architecture for Vision
• Pay Attention to MLPs (gMLP)
• Vision Transformer (ViT)
• Primer EZ
• Hourglass

• Generate on a 48GB GPU
• Finetune on two 48GB GPUs
• LLM.int8()

• Denoising Diffusion Probabilistic Models (DDPM)
• Denoising Diffusion Implicit Models (DDIM)
• Latent Diffusion Models
• Stable Diffusion

• Original GAN
• GAN with deep convolutional network
• Cycle GAN
• Wasserstein GAN
• Wasserstein GAN with Gradient Penalty
• StyleGAN 2

• Graph Attention Networks (GAT)
• Graph Attention Networks v2 (GATv2)