ModelVérifiéExterne

Dynrank Compressed Models

Hardened SVD-compressed LLM checkpoints from the DynRank paper: "DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"

Sourcehuggingface.co/zhc12/dynrank-compressed-models/tree/fdb71269f50834f26b85e0ba49c48f73555bf92a

TypeOtherModèle de baseQwen3VersionRevision fdb7126Éditeur@zhc12CClasse

Source du modèle

Type: Other
Taille du fichier: 5,80–24,1 GB
Modèle de base: Qwen3
Version: vRevision fdb7126
Mis à jour: 2 oct. 2026
Éditeur: @zhc12
Téléchargements: 0
Paramètres: 7B
Source: Hugging Face

Description de la source

DynRank: Compressed Model Checkpoints

Hardened SVD-compressed LLM checkpoints from the DynRank paper: "DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"

Checkpoint Format

Each .pt file is a dictionary:

{
    "step": 1000,
    "ranks": [r_1, r_2, ..., r_224],  # per-sublayer integer ranks
    "state_dict": {...},               # model weights (CompressedLinear format)
    "val_loss": float
}

Weight keys use CompressedLinear format:

model.layers.{i}.self_attn.q_proj.first.weight — B matrix (r × n)
model.layers.{i}.self_attn.q_proj.second.weight — A matrix (d × r)

Loading

import torch
ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu")
ranks = ckpt["ranks"]        # list of 224 integers
state_dict = ckpt["state_dict"]

To use with SVD-LLM's CompressedLinear:

from src.model.replace import CompressedLinear
# See https://github.com/zhc1212/SVD-LLM for full loading code

Directory Structure

baselines/           # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2)
phase5a/             # Early TinyLlama logs (no model weights)
phase5b_llama2/      # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B)
phase5e_qwen3/       # Qwen3-8B experiments

phase5b_llama2 key models

Directory	Model	Method	Ratio	Notes
`v9_r04`	LLaMA-2-7B	DynRank	0.4	Primary result (seed=42)
`static_r04`	LLaMA-2-7B	Static uniform	0.4	Matched baseline
`v9_r04_s3/s4_refill/s5`	LLaMA-2-7B	DynRank	0.4	Multi-seed (43,44,47)
`v9_r02/r08`	LLaMA-2-7B	DynRank	0.2/0.8	Ratio sweep
`llama7b_v9_r04`	LLaMA-7B	DynRank	0.4	Cross-model
`mistral_v9_r04`	Mistral-7B	DynRank	0.4	Cross-model
`llama2_13b_v9_r04`	LLaMA-2-13B	DynRank	0.4	Scale test
`baseline_dobi_r04`	LLaMA-2-7B	Dobi-SVD profile	0.4	Baseline comparison
`baseline_asvd_r04_v2`	LLaMA-2-7B	ASVD-STRS profile	0.4	Baseline comparison
`h2_oracle_llama2_r04`	LLaMA-2-7B	Oracle static	0.4	H2 ablation
`gsm8k_dynrank_r04`	LLaMA-2-7B	DynRank (GSM8K calib)	0.4	Domain analysis

JSON files

cross_eval_*.json — perplexity evaluation results (WT2, C4, LAMBADA) downstream7_*.json — zero-shot accuracy on 6 tasks

Citation

@article{zhang2026dynrank,
  title={DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression},
  author={Zhang, Huicheng},
  year={2026}
}

License

Model weights are derivatives of:

LLaMA / LLaMA-2 (Meta Community License)
Mistral-7B (Apache 2.0)
Qwen3-8B (Qwen License)

Use subject to the original model licenses.

Sources

1 source

Hugging FacePrincipalVersion fdb71269f50834f26b85e0ba49c48f73555bf92a

huggingface.co/zhc12/dynrank-compressed-models/tree/fdb71269f50834f26b85e0ba49c48f73555bf92a

Vérifié 2 oct.

Artefacts du modèle

64 artefacts

batch1_r439_r438_20260727/r04_s43_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 a835d52ed9fc…72e7 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s43_own/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 fc6ea2ea88a7…5ae3 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s43_perm-pooled/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 8cb4c2f85056…22d9 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s43_perm-typeswap/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 eb34def1c237…b542 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s44_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 b117c1889cb6…3770 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s44_own/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 a75d9a0f4c9e…cc99 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s44_perm-pooled/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 4eefeff0c975…1296 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s44_perm-typeswap/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 023c2eccffd8…ad8f · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s47_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 c06b5357acfd…53b9 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s47_own/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 8937f3c5412a…6f7a · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s47_perm-pooled/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 629c9316cd90…ada7 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r04_s47_perm-typeswap/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 30792f4197e2…1bfe · Hugging Face

Télécharger

batch1_r439_r438_20260727/r06_s48_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 f2bf1d5dd73c…2cff · Hugging Face

Télécharger

batch1_r439_r438_20260727/r06_s49_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 5033939337b1…b0d7 · Hugging Face

Télécharger

batch1_r439_r438_20260727/r06_s50_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 3a01bb984942…6206 · Hugging Face

Télécharger

dobi_paired_20260725/r04_s43_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 a835d52ed9fc…72e7 · Hugging Face

Télécharger

dobi_paired_20260725/r04_s44_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 d323777d5028…9b9a · Hugging Face

Télécharger

dobi_paired_20260725/r04_s47_dobi/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 75cd82e5d140…fe09 · Hugging Face

Télécharger

dobi_paired_20260725/r06_s48_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 1f90810ed928…90e1 · Hugging Face

Télécharger

dobi_paired_20260725/r06_s49_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 7354a64e9245…c9b1 · Hugging Face

Télécharger

dobi_paired_20260725/r06_s50_dobi/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 3a01bb984942…6206 · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_const10000/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 95b5d664fb7b…5c77 · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_const2000/v8_hardened_step1000.pt

pt · 15,5 GB · SHA-256 e4b8243f0693…9301 · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_const50000/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 4cb42c168817…ae4f · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_lam100/v8_hardened_step1000.pt

pt · 15,8 GB · SHA-256 b6c05376c9c4…167f · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_lam2000/v8_best_soft.pt

pt · 24,1 GB · SHA-256 57b0aefc22f6…73fc · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_lam2000/v8_hardened_step1000.pt

pt · 10,0 GB · SHA-256 497f7df2cece…4d59 · Hugging Face

Télécharger

e2_ablation/penalty_r0p4_lam500/v8_hardened_step1000.pt

pt · 15,8 GB · SHA-256 b3d924dfa799…11d0 · Hugging Face

Télécharger

h2_oracle_matched_20260725/r04_s43_oracle/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 fc6ea2ea88a7…5ae3 · Hugging Face

Télécharger

h2_oracle_matched_20260725/r04_s44_oracle/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 1535ef0ae434…4765 · Hugging Face

Télécharger

h2_oracle_matched_20260725/r04_s47_oracle/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 8937f3c5412a…6f7a · Hugging Face

Télécharger

h2_oracle_matched_20260725/r06_s48_oracle/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 c0a6fcb09bab…dcb3 · Hugging Face

Télécharger

h2_oracle_matched_20260725/r06_s49_oracle/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 dd3eaaaaf8fd…0475 · Hugging Face

Télécharger

h2_oracle_matched_20260725/r06_s50_oracle/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 4ee5112fe335…39c7 · Hugging Face

Télécharger

phase5b_llama2_ft/static_r02/v8_hardened_step1000.pt

pt · 19,8 GB · SHA-256 4a0d3455a8a0…69ee · Hugging Face

Télécharger

phase5b_llama2_ft/static_r04_s3/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 7b272e677ccc…3ae2 · Hugging Face

Télécharger

phase5b_llama2_ft/static_r06_s4/v8_hardened_step1000.pt

pt · 10,1 GB · SHA-256 f6c0174f8e21…4f7c · Hugging Face

Télécharger

phase5b_llama2_ft/v9_r02_vi10/v8_hardened_step1000.pt

pt · 20,3 GB · SHA-256 c027cc2667f4…ed4c · Hugging Face

Télécharger

phase5b_llama2_ft/v9_r02/v8_hardened_step1000.pt

pt · 19,8 GB · SHA-256 e353f0813943…6cf7 · Hugging Face

Télécharger

phase5b_llama2_ft/v9_r04_s3/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 0e5d45ccf11f…6ff9 · Hugging Face

Télécharger

phase5b_llama2_ft/v9_r06_s4/v8_hardened_step1000.pt

pt · 10,1 GB · SHA-256 00d51da1f9ec…7627 · Hugging Face

Télécharger

phase5b_llama2/ablation_bf16_run2_r04/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 d9965eb85c41…387c · Hugging Face

Télécharger

phase5b_llama2/ablation_calib128_r04/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 7d9ba4d6914c…18a2 · Hugging Face

Télécharger

phase5b_llama2/ablation_calib512_r04/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 c8cd92beb656…7cab · Hugging Face

Télécharger

phase5b_llama2/ablation_calib64_r04/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 76bf54edf742…79b0 · Hugging Face

Télécharger

phase5b_llama2/ablation_tau10_r04/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 945f426dc272…0f85 · Hugging Face

Télécharger

phase5b_llama2/ablation_tau50_r04/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 cfba39380c3e…85ee · Hugging Face

Télécharger

phase5b_llama2/baseline_dobi_r04/v8_hardened_step1000.pt

pt · 15,0 GB · SHA-256 f6e1002936a9…784a · Hugging Face

Télécharger

phase5b_llama2/baseline_dobi_r06/v8_hardened_step1000.pt

pt · 10,1 GB · SHA-256 87af2dda0d88…f3eb · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_p20_s1/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 c8ad24f002cf…cf97 · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_p20_s2/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 11cf9848b652…c9cc · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_p20_s42/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 fdd344c789cc…67bc · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_s1/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 73dcd1abcad4…bd4f · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_s2/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 d2b79d7cb281…bab8 · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r04_s42/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 b8e647e92a97…65ce · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r06_s42/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 5af7bf9bc504…1acc · Hugging Face

Télécharger

phase5b_llama2/code_dynrank_r08_s42/v8_hardened_step1000.pt

pt · 5,80 GB · SHA-256 5f755ca27c9e…da0d · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_p20_s1/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 7850a489bc70…dea7 · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_p20_s2/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 4c68d32e3c4b…310c · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_p20_s42/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 de1c3e697d21…7327 · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_s1/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 8c1f1d3e43dd…7a2d · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_s2/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 7512147b9be2…7902 · Hugging Face

Télécharger

phase5b_llama2/code_static_r04_s42/v8_hardened_step1000.pt

pt · 15,4 GB · SHA-256 9ce393e4b3cd…151d · Hugging Face

Télécharger

phase5b_llama2/code_static_r06_s42/v8_hardened_step1000.pt

pt · 10,6 GB · SHA-256 70dc9e1c858e…1ab9 · Hugging Face

Télécharger

Extraits de sources

3 extraits

Readme11 juil.huggingface.co/zhc12/dynrank-compressed-models

# DynRank: Compressed Model Checkpoints Hardened SVD-compressed LLM checkpoints from the DynRank paper: **"DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"** ## Checkpoint Format Each `.pt` file is a dictionary: ```python { "step": 1000, "ranks": [r_1, r_2, ..., r_224], # per-sublayer integer ranks "state_dict": {...}, # model weights (CompressedLinear format) "val_loss": float } ``` Weight keys use `CompressedLinear` format: - `model.layers.{i}.self_attn.q_proj.first.weight` — B matrix (r × n) - `model.layers.{i}.self_attn.q_proj.second.weight` — A matrix (d × r) ## Loading ```python import torch ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu") ranks = ckpt["ranks"] # list of 224 integers state_dict = ckpt["state_dict"] ``` To use with SVD-LLM's `CompressedLinear`: ```python from src.model.replace import CompressedLinear # See https://github.com/zhc1212/SVD-LLM for full loading code ``` ## Directory Structure ``` baselines/ # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2) phase5a/ # Early TinyLlama logs (no model weights) phase5b_llama2/ # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B) phase5e_qwen3/ # Qwen3-8B experiments ``` ### phase5b_llama2 key models | Directory | Model | Method | Ratio | Notes | |-----------|-------|--------|-------|-------| | `v9_r04` | LLaMA-2-7B | DynRank | 0.4 | Primary result (seed=42) | | `static_r04` | LLaMA-2-7B | Static uniform | 0.4 | Matched baseline | | `v9_r04_s3/s4_refill/s5` | LLaMA-2-7B | DynRank | 0.4 | Multi-seed (43,44,47) | | `v9_r02/r08` | LLaMA-2-7B | DynRank | 0.2/0.8 | Ratio sweep | | `llama7b_v9_r04` | LLaMA-7B | DynRank | 0.4 | Cross-model | | `mistral_v9_r04` | Mistral-7B | DynRank | 0.4 | Cross-model | | `llama2_13b_v9_r04` | LLaMA-2-13B | DynRank | 0.4 | Scale test | | `baseline_dobi_r04` | LLaMA-2-7B | Dobi-SVD profile | 0.4 | Baseline comparison | | `baseline_asvd_r04_v2` | LLaMA-2-7B | ASVD-STRS profile | 0.4 | Baseline comparison | | `h2_oracle_llama2_r04` | LLaMA-2-7B | Oracle static | 0.4 | H2 ablation | | `gsm8k_dynrank_r04` | LLaMA-2-7B | DynRank (GSM8K calib) | 0.4 | Domain analysis | ### JSON files `cross_eval_*.json` — perplexity evaluation results (WT2, C4, LAMBADA) `downstream7_*.json` — zero-shot accuracy on 6 tasks ## Citation ```bibtex @article{zhang2026dynrank, title={DynRank...

Note11 juil.huggingface.co/zhc12/dynrank-compressed-models

Source context: 0 downloads · 0 likes · Repo zhc12/dynrank-compressed-models

Note18 maihuggingface.co/api/models/zhc12/dynrank-compressed-models

Source context: 0 downloads · 0 likes · Repo zhc12/dynrank-compressed-models