# DynRank: Compressed Model Checkpoints Hardened SVD-compressed LLM checkpoints from the DynRank paper: **"DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"** ## Checkpoint Format Each `.pt` file is a dictionary: ```python { "step": 1000, "ranks": [r_1, r_2, ..., r_224], # per-sublayer integer ranks "state_dict": {...}, # model weights (CompressedLinear format) "val_loss": float } ``` Weight keys use `CompressedLinear` format: - `model.layers.{i}.self_attn.q_proj.first.weight` — B matrix (r × n) - `model.layers.{i}.self_attn.q_proj.second.weight` — A matrix (d × r) ## Loading ```python import torch ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu") ranks = ckpt["ranks"] # list of 224 integers state_dict = ckpt["state_dict"] ``` To use with SVD-LLM's `CompressedLinear`: ```python from src.model.replace import CompressedLinear # See https://github.com/zhc1212/SVD-LLM for full loading code ``` ## Directory Structure ``` baselines/ # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2) phase5a/ # Early TinyLlama logs (no model weights) phase5b_llama2/ # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B) phase5e_qwen3/ # Qwen3-8B experiments ``` ### phase5b_llama2 key models | Directory | Model | Method | Ratio | Notes | |-----------|-------|--------|-------|-------| | `v9_r04` | LLaMA-2-7B | DynRank | 0.4 | Primary result (seed=42) | | `static_r04` | LLaMA-2-7B | Static uniform | 0.4 | Matched baseline | | `v9_r04_s3/s4_refill/s5` | LLaMA-2-7B | DynRank | 0.4 | Multi-seed (43,44,47) | | `v9_r02/r08` | LLaMA-2-7B | DynRank | 0.2/0.8 | Ratio sweep | | `llama7b_v9_r04` | LLaMA-7B | DynRank | 0.4 | Cross-model | | `mistral_v9_r04` | Mistral-7B | DynRank | 0.4 | Cross-model | | `llama2_13b_v9_r04` | LLaMA-2-13B | DynRank | 0.4 | Scale test | | `baseline_dobi_r04` | LLaMA-2-7B | Dobi-SVD profile | 0.4 | Baseline comparison | | `baseline_asvd_r04_v2` | LLaMA-2-7B | ASVD-STRS profile | 0.4 | Baseline comparison | | `h2_oracle_llama2_r04` | LLaMA-2-7B | Oracle static | 0.4 | H2 ablation | | `gsm8k_dynrank_r04` | LLaMA-2-7B | DynRank (GSM8K calib) | 0.4 | Domain analysis | ### JSON files `cross_eval_*.json` — perplexity evaluation results (WT2, C4, LAMBADA) `downstream7_*.json` — zero-shot accuracy on 6 tasks ## Citation ```bibtex @article{zhang2026dynrank, title={DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression}, author={Zhang, Huicheng}, year={2026} } ``` ## License Model weights are derivatives of: - LLaMA / LLaMA-2 (Meta Community License) - Mistral-7B (Apache 2.0) - Qwen3-8B (Qwen License) Use subject to the original model licenses.
