Skip to content
ModelListedExternal
Browse files

RetFormerTrainedOnHDMB51

RetFormer is a hybrid video classification model that replaces the temporal attention in TimeSformer with RetNet, achieving:

Sourcehuggingface.co/sumit7488/RetFormerTrainedOnHDMB51/tree/a42b837ea40b280d83af06b3fff8c09fcf188ef2
KindOtherVersionModelBgrade

Model source

Kind
Other
File size
494 MB
Version
vModel
Updated
Oct 24, 2026
Source
Hugging Face

Source description

🎬 RetFormer: Efficient TimeSformer + RetNet for Video Action Recognition

RetFormer is a hybrid video classification model that replaces the temporal attention in TimeSformer with RetNet, achieving:

  • ⚡ Lower memory usage
  • 🚀 Faster training
  • 🎯 Competitive accuracy

🧠 Model Architecture

🔹 RetFormer (Proposed)

  • Spatial Modeling → TimeSformer
  • Temporal Modeling → RetNet

👉 This replaces quadratic attention with linear-time temporal modeling (O(n))

📊 Dataset

  • HMDB51
    • 51 human action classes
    • Complex motion patterns
    • Smaller and more challenging than UCF101

🔁 Training Strategy

Training was performed in multiple stages due to runtime limits:

  • Initial training (Epoch 1–10)
  • Checkpoint saving
  • Resumed training (Epoch 11–14)
  • Early stopping applied

📈 Training Results (Epoch 1–14)

EpochTrain LossTrain AccVal LossVal AccF1
13.93120.03503.80990.09670.0855
23.63300.17913.29480.36540.3149
33.09890.36912.69270.51500.4579
42.62780.50482.28790.58690.5503
52.31980.57822.04380.62550.5961
62.13870.61941.91520.62420.6074
71.98760.66571.83690.64180.6308
81.91400.69361.79660.63590.6188
91.85390.70411.76190.65560.6426
101.81490.72441.75230.66140.6512
111.73250.75241.73150.66990.6614
121.70360.75841.74690.66210.6515
131.66820.77171.75040.65950.6496
141.63440.77851.74880.65880.6494

🏆 Best Performance

  • Validation Accuracy: 66.99%
  • F1 Score: 0.6614
  • Achieved at Epoch 11

⚙️ Training Details

  • Peak GPU Memory: ~7.2 GB
  • Training Time per Epoch: ~52 minutes
  • Evaluation Time: ~8 minutes
  • Mixed Precision Training (torch.cuda.amp)
  • Early stopping triggered after Epoch 14

📌 Observations

  • Stable improvement until Epoch 11
  • Slight decline afterward → early overfitting
  • Lower accuracy than baseline (expected for hybrid trade-off)

⚡ Efficiency Advantage

MetricTimeSformerRetFormer
Peak GPU Memory~9.3 GB~7.2 GB ✅
ComplexityO(n2)O(n) ✅
SpeedSlowerFaster

👉 ~25% reduction in GPU memory

🔍 Key Insight

RetFormer demonstrates that:

  • Efficient temporal modeling can significantly reduce memory usage
  • Performance remains competitive with baseline models
  • Trade-off exists between efficiency and maximum accuracy

🚀 Usage

pip install torch torchvision transformers

Sources

1 source
Hugging FacePrimaryVersion a42b837ea40b280d83af06b3fff8c09fcf188ef2
huggingface.co/sumit7488/RetFormerTrainedOnHDMB51/tree/a42b837ea40b280d83af06b3fff8c09fcf188ef2

Verified Oct 24

Download files

Model artifacts

2 artifacts

model.safetensors

safetensors · 494 MB · SHA-256 7840c5ac2533…6015 · Hugging Face

Download file

pytorch_model.bin

bin · 494 MB · SHA-256 73a23ebdbf66…b967 · Hugging Face

Download file

Source excerpts

2 excerpts
ReadmeAug 6huggingface.co/sumit7488/RetFormerTrainedOnHDMB51

--- license: apache-2.0 tags: - video-classification - timesformer - retnet - action-recognition - hmdb51 - efficient-models - transformers datasets: - hmdb51 --- # 🎬 RetFormer: Efficient TimeSformer + RetNet for Video Action Recognition RetFormer is a hybrid video classification model that replaces the **temporal attention** in TimeSformer with **RetNet**, achieving: - ⚡ Lower memory usage - 🚀 Faster training - 🎯 Competitive accuracy --- ## 🧠 Model Architecture ### 🔹 RetFormer (Proposed) - Spatial Modeling → TimeSformer - Temporal Modeling → **RetNet** 👉 This replaces quadratic attention with **linear-time temporal modeling (O(n))** --- ## 📊 Dataset - **HMDB51** - 51 human action classes - Complex motion patterns - Smaller and more challenging than UCF101 --- ## 🔁 Training Strategy Training was performed in multiple stages due to runtime limits: - Initial training (Epoch 1–10) - Checkpoint saving - Resumed training (Epoch 11–14) - Early stopping applied --- ## 📈 Training Results (Epoch 1–14) | Epoch | Train Loss | Train Acc | Val Loss | Val Acc | F1 | |------|------------|-----------|----------|---------|-----| | 1 | 3.9312 | 0.0350 | 3.8099 | 0.0967 | 0.0855 | | 2 | 3.6330 | 0.1791 | 3.2948 | 0.3654 | 0.3149 | | 3 | 3.0989 | 0.3691 | 2.6927 | 0.5150 | 0.4579 | | 4 | 2.6278 | 0.5048 | 2.2879 | 0.5869 | 0.5503 | | 5 | 2.3198 | 0.5782 | 2.0438 | 0.6255 | 0.5961 | | 6 | 2.1387 | 0.6194 | 1.9152 | 0.6242 | 0.6074 | | 7 | 1.9876 | 0.6657 | 1.8369 | 0.6418 | 0.6308 | | 8 | 1.9140 | 0.6936 | 1.7966 | 0.6359 | 0.6188 | | 9 | 1.8539 | 0.7041 | 1.7619 | 0.6556 | 0.6426 | | 10 | 1.8149 | 0.7244 | 1.7523 | 0.6614 | 0.6512 | | 11 | 1.7325 | 0.7524 | 1.7315 | **0.6699** | **0.6614** | | 12 | 1.7036 | 0.7584 | 1.7469 | 0.6621 | 0.6515 | | 13 | 1.6682 | 0.7717 | 1.7504 | 0.6595 | 0.6496 | | 14 | 1.6344 | 0.7785 | 1.7488 | 0.6588 | 0.6494 | --- ## 🏆 Best Performance - **Validation Accuracy:** **66.99%** - **F1 Score:** 0.6614 - Achieved at **Epoch 11** --- ## ⚙️ Training Details - Peak GPU Memory: **~7.2 GB** - Training Time per Epoch: ~52 minutes - Evaluation Time: ~8 minutes - Mixed Precision Training (`torch.cuda.amp`) - Early stopping triggered after Epoch 14 --- ## 📌 Observations - Stable improvement until **Epoch 11** - Slight decline afterward → early overfitting - Lower accuracy than baseline (expected for hybrid trade-off)...

NoteAug 6huggingface.co/sumit7488/RetFormerTrainedOnHDMB51

Source context: 3 downloads · 0 likes · Pipeline video-classification · Library transformers · Repo sumit7488/RetFormerTrainedOnHDMB51