--- language: en license: mit tags: - deep-learning - resnet - celebA - pytorch - from-scratch --- # DL From Scratch Implement mainstream deep learning models from scratch. ## Project Structure ``` ├── main.py ├── pyproject.toml ├── .gitignore ├── README.md ├── ROADMAP.md ├── ml/ # Classical Machine Learning (pure NumPy) │ ├── mlp/ # MLP (MNIST, manual backprop) │ └── basics/ # 12 standalone models (lin/log reg, SVM, K-Means, PCA, RF, GBDT, etc.) ├── cv/ # Computer Vision │ ├── simplecnn/ # SimpleCNN (CIFAR-10, Conv×3+Pool×3+FC×2) │ ├── resnet18/ # ResNet18 (CelebA, 15 attrs, skip connections) │ ├── resnet34/ # ResNet34 (CelebA, 40 attrs, [3,4,6,3] blocks) │ ├── resnet50/ # ResNet50 (Bottleneck block 1×1→3×3→1×1) │ ├── mobilenet/ # MobileNet (depthwise separable conv, CIFAR-10) │ ├── vit/ # Vision Transformer (patch embed + BERT encoder, CIFAR-10) │ ├── unet/ # UNet (Oxford-IIIT Pet segmentation) │ └── yolo/ # YOLO (Pascal VOC object detection) ├── gen/ # Generative Models │ ├── dcgan/ # DCGAN (CelebA, transposed conv) │ ├── vae/ # VAE (reparameterization trick, KL divergence) │ ├── ddpm/ # DDPM (CIFAR-10, denoising diffusion) │ └── simclr/ # SimCLR (CIFAR-10, contrastive learning) ├── graph/ # Graph Neural Networks │ └── gcn/ # GCN (Cora, spectral graph convolution) ├── rl/ # Reinforcement Learning │ └── dqn/ # DQN (CartPole, experience replay) ├── nlp/ # Natural Language Processing │ ├── bert/ # BERT (MLM pretrain + classification finetune) │ ├── gpt/ # GPT (decoder-only, causal attention, KV cache) │ ├── lstm/ # LSTM (hand-written gates, IMDB sentiment) │ ├── word2vec/ # Word2Vec (CBOW + Skip-gram, negative sampling) │ ├── seq2seq/ # Seq2Seq Transformer (EN→DE translation) │ └── lora/ # LoRA (parameter-efficient GPT fine-tuning) ├── utils/ # Shared Infrastructure │ ├── config.py # YAML config loading/saving │ ├── seed.py # Reproducibility seed locking │ └── device.py # CUDA → MPS → CPU auto-detection └── scripts/ # Notebook generation scripts │ ├── __init__.py │ ├── config.yaml # DCGAN hyperparameters │ ├── model.py # Generator + Discriminator │ ├── data.py # CelebA images (64×64, no labels) │ ├── train.py # Adversarial training loop (G/D alternating) │ └── generate.py # Generate sample grid from trained model ├── vit/ │ ├── __init__.py │ ├── config.yaml # ViT hyperparameters (patch_size, d_model, n_layers, etc.) │ ├── model.py # ViT: PatchEmbed → Transformer encoder (reused from BERT) → CLS head │ ├── data.py # CIFAR-10 via HF datasets │ ├── train.py # Training loop │ └── eval.py # Per-class accuracy on test split ├── unet/ │ ├── __init__.py │ ├── config.yaml # UNet hyperparameters │ ├── model.py # U-Net: encoder–decoder with skip connections │ ├── data.py # Oxford-IIIT Pet (image + mask) with augmentation │ ├── train.py # Training loop (pixel-wise CrossEntropy) │ └── eval.py # IoU and pixel accuracy ├── cnn/ │ ├── __init__.py │ ├── data.py # CIFAR-10 via HF datasets (uoft-cs/cifar10) │ ├── model.py # Plain CNN (Conv×3 + Pool×3 + FC×2) │ ├── train.py # Training script (Adam + CosineAnnealingLR) │ └── eval.py # Test evaluation + confusion matrix ├── mlp/ │ ├── __init__.py │ ├── data.py # MNIST via HF datasets (ylecun/mnist) │ ├── model.py # MLP — pure NumPy (Linear, ReLU, SoftmaxCrossEntropy, SGD) │ ├── train.py # Training script │ └── eval.py # Test evaluation (per-digit accuracy) ├── utils/ │ ├── __init__.py │ ├── config.py # YAML config loading/saving (load_config / save_config) │ └── seed.py # set_seed() — lock torch + numpy + random + cudnn ├── nlp/ │ ├── bert/ │ ├── word2vec/ │ ├── lstm/ │ ├── gpt/ │ └── seq2seq/ │ ├── __init__.py │ ├── tokenizer.py # Word-level tokenizer (5000 vocab, from text8) │ ├── model.py # Decoder-only Transformer (Causal Attention + KV Cache) │ ├── train.py # Autoregressive LM on text8 │ └── generate.py # Text generation (temperature + top-k + [SEP] blocked) │ └── seq2seq/ │ ├── __init__.py │ ├── config.yaml # Transformer hyperparameters │ ├── model.py # Encoder (from BERT) + Decoder (cross-attention) → Seq2Seq │ ├── data.py # Multi30k EN→DE, word-level tokenizer │ ├── train.py # Teacher forcing training │ └── generate.py # Greedy decoding translation demo ├── basics/ │ ├── __init__.py │ ├── logistic_regression.py # Single Linear layer + Softmax (92.3% on MNIST) │ ├── linear_regression.py # California Housing (Normal Equation + GD, R2=0.583) │ ├── k_means.py # Unsupervised clustering (pure NumPy) │ ├── svm.py # SVM — GD (primal) + SMO (dual, Linear/RBF kernels) │ ├── decision_tree.py # ID3/CART on Iris (ASCII tree, ~93% acc) │ ├── random_forest.py # Bagging + random feature subsets, Iris 93.3% │ ├── gbdt.py # Gradient boosting (MSE reg + binary logloss cls) │ ├── naive_bayes.py # Gaussian NB on MNIST (generative classifier) │ ├── pca.py # SVD-based dimensionality reduction (MNIST 2D visualisation) │ ├── knn.py # k-Nearest Neighbors (instance-based, MNIST) │ └── perceptron.py # Single neuron (Rosenblatt 1958, step activation) ├── .gitattributes # LFS: *.zip *.pt └── uv.lock ``` ## Infrastructure | Feature | Description | |---|---| | **Config system** | Each model directory has a `config.yaml` with its hyperparameters (seed, lr, batch_size, epochs, etc.). Edit the YAML to change training params without touching code. | | **TensorBoard** | Every PyTorch training script logs loss/accuracy per epoch to `runs/{model_name}/`. Run `tensorboard --logdir runs` to visualize all experiments. | | **Reproducibility** | `utils/seed.py` provides `set_seed()` that locks `torch` + `numpy` + `random` + `cudnn`. Called at the start of every train script. Config is saved alongside model weights (`_config.yaml`). | ### Usage ```bash # View training curves (all models) tensorboard --logdir runs # Edit hyperparameters in YAML instead of code vim cv/resnet18/config.yaml # then train as usual: uv run python -m cv.resnet18.train ``` ## CV/ResNet18 | Item | Value | |---|---| | Model | ResNet18 (11.2M params) | | Dataset | CelebA via HF datasets — 1,000 images | | Attributes | 15 binary (Smiling, Male, Young, Eyeglasses, etc.) | | Split | 800 train / 200 val | | Val Accuracy | **91.2%** | | Training | MPS (Mac M4) + AMP | ## CV/ResNet34 | Item | Value | |---|---| | Model | ResNet34 (~21M params, [3,4,6,3] BasicBlock) | | Dataset | CelebA via HF datasets — full 200K | | Attributes | All 40 binary attributes | | Optimizer | SGD + Momentum (0.9, weight_decay=1e-4) | | Training | CosineAnnealingLR + Gradient Accumulation + Early Stopping + Loss Weighting | ## CV/ResNet50 | Item | Value | |---|---| | Model | ResNet50 (~23.6M params, [3,4,6,3] Bottleneck) | | Dataset | CelebA via HF datasets — full 200K | | Attributes | All 40 binary attributes | | Optimizer | SGD + Momentum (0.9, weight_decay=1e-4) | | Architecture | Bottleneck block: 1×1 → 3×3 → 1×1 (contrast with BasicBlock's two 3×3) | ## GEN/VAE | Item | Value | |---|---| | Model | Variational Autoencoder (2.6M params) | | Dataset | CelebA via HF datasets — 10K images (64×64) | | Architecture | Conv Encoder → μ,logσ2 → reparameterize → Deconv Decoder → Sigmoid | | Loss | Reconstruction (BCE) + KL divergence | | Training | Adam(lr=2e-4), 50 epoch | ## NLP/Seq2Seq Transformer | Item | Value | |---|---| | Model | Encoder-Decoder Transformer (1M params) | | Dataset | Multi30k EN→DE — 29K train / 1K test | | Architecture | Encoder (from BERT) + Decoder (causal + cross-attention) | | Training | Teacher forcing, weight-tying, Adam(lr=1e-4) | ## GEN/DDPM | Item | Value | |---|---| | Model | Denoising Diffusion (16.1M params) | | Dataset | CIFAR-10 via HF datasets — 50K images (32×32) | | Architecture | UNet + timestep embedding + sinusoid positional encoding | | Training | Noise prediction (MSE), T=1000, linear β schedule | | Sampling | Reverse diffusion (x_T → x_0), 1000 steps | ## GRAPH/GCN | Item | Value | |---|---| | Model | 2-layer Graph Convolutional Network (23K params) | | Dataset | Cora via URL — 2708 nodes, 1433 features, 7 classes | | Architecture | GraphConv × 2: Â @ H @ W (spectral graph convolution) | | Training | Semi-supervised (20 labels/class), CrossEntropyLoss | ## RL/DQN | Item | Value | |---|---| | Model | Deep Q-Network (17K params) | | Environment | CartPole-v1 via Gymnasium — 4-dim state, 2 actions | | Architecture | 3-layer MLP (4→128→128→2) | | Training | Experience replay, target network, ε-greedy decay | ## GEN/SimCLR | Item | Value | |---|---| | Model | SimCLR (11M params: ResNet18 encoder + MLP projector) | | Dataset | CIFAR-10 via HF datasets — self-supervised (no labels) | | Architecture | ResNet18 → Projector(512→256→128) → NT-Xent loss | | Training | 100 epoch, temperature=0.5, dual random augmentation | ## CV/YOLO | Item | Value | |---|---| | Model | Simplified YOLO (59M params) | ## NLP/LoRA | Item | Value | |---|---| | Model | Low-Rank Adaptation on GPT (32K trainable / 5.7M frozen) | | Dataset | text8 via HF datasets — 5K chunks | | Architecture | LoRALayer: frozen Linear + low-rank B×A | | Key concept | Parameter-efficient fine-tuning, 0.58% trainable params | | Comparison | Full fine-tune: 5.7M vs LoRA r=8: 32K | ## CV/MobileNet | Item | Value | |---|---| | Model | MobileNetV1 (135K params, width=1.0) | | Dataset | CIFAR-10 via HF datasets — 50K train / 10K test | | Architecture | DepthwiseSeparableConv (depthwise 3×3 + pointwise 1×1) | | Key concept | Depthwise separable convolution, ~8.4× fewer ops than standard conv | | Comparison | SimpleCNN 620K params → MobileNet 135K (4.6× smaller) | | Dataset | Pascal VOC via HF datasets — 20 classes | | Architecture | CNN backbone → FC detection head → 7×7×30 output | | Training | YOLO loss (coord + obj + noobj + class), NMS at inference | ## DCGAN | Item | Value | |---|---| | Model | Generator (3.5M params) + Discriminator (2.8M params) | | Dataset | CelebA via HF datasets — 10K images (64×64) | | Architecture | Transposed conv G / Conv D, BN, LeakyReLU | | Optimizer | Adam(lr=2e-4, β1=0.5) — separate for G and D | | Training | BCELoss, label smoothing, fixed noise grid for monitoring | ## CV/ViT | Item | Value | |---|---| | Model | Vision Transformer (807K params, 4 layers, 4 heads, 128-dim) | | Dataset | CIFAR-10 via HF datasets — 50K train / 10K test | | Architecture | PatchEmbed(4×4) → [CLS] → Transformer Encoder (from BERT) → CLS head | | Key concept | Self-attention for vision, no convolutions, patch embeddings | ## CV/UNet | Item | Value | |---|---| | Model | U-Net (31M params, 5 encoder/decoder stages) | | Dataset | Oxford-IIIT Pet via HF datasets — image + segmentation mask | | Architecture | Encoder: Conv+MaxPool × 4, Decoder: UpConv+skip × 4, output: pixel-wise logits | | Loss | CrossEntropy (ignore_index=0 for unlabeled) | | Metrics | Pixel accuracy, mean IoU | ## CV/SimpleCNN | Item | Value | |---|---| | Model | SimpleCNN (620K params) | | Dataset | CIFAR-10 via HF datasets — 50K images | | Classes | 10 (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck) | | Test Accuracy | **82.4%** (30 epochs) | | Training | Adam + CosineAnnealingLR | ## ML/MLP | Item | Value | |---|---| | Model | MLP (235K params, pure NumPy) | | Dataset | MNIST via HF datasets — 60K images | | Classes | 10 digits (0-9) | | Test Accuracy | **97.9%** (20 epochs) | | Framework | NumPy only (hand-written backward pass) | ## BERT | Item | Value | |---|---| | Model | BERT mini (834K params, 4 layers, 4 heads, 128-dim) | | Pre-training | MLM on text8 (90M chars, HuggingFace) | | Fine-tuning | Sentiment classification on IMDB (HuggingFace) | | Test Accuracy | ~50% (character-level; word-level would be higher with subword tokenization) | | Core components | Self-Attention (semantic aggregation) + MLM (entropy increase noise reduction) | ## Word2Vec | Item | Value | |---|---| | Model | Word2Vec (50-dim embeddings, 97K vocab) | | Architectures | CBOW + Skip-gram with Negative Sampling | | Dataset | text8 via HF datasets (~90M chars) | | Training | Adam, 5 epochs, k=5 negative samples | | Evaluation |...
