Escarda VE

--- license: apache-2.0 tags: - vision-encoder - jepa - hrm - dinov2-distillation --- # Escarda-VE Vision encoder for the Escarda family (~39.6M, 448px / patch16 / 784 tokens). **Byrne-VE** plus a **JEPA head next to the HRM refine block** - that's the Escarda trait. - ViT-style: RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM refine. - **JEPA head** (auxiliary): from a patch token, predict the raster-order neighbour patch's representation (stop-gradient target, 1-cosine loss). **Zero inference cost.** - Distilled from frozen **DINOv2-base** (50k steps, CLS + patch), then 20k steps of DINO-style **self-distillation** (EMA teacher, cosine prototypes, no collapse). ## Escarda-VE vs Byrne-VE (DINOv2 teacher-alignment, n=1024 held-out) | | Byrne-VE | Escarda-VE | |---|---|---| | Params | 39.34M | 39.60M (+JEPA head) | | CLS cosine | **0.776** | 0.771 | | PATCH cosine | **0.600** | 0.584 | | JEPA self-consistency | - | 0.040 | Escarda trades ~1-3% teacher-alignment (JEPA pulls a bit of capacity off pure mimicry) for a self-supervised spatial neighbour-prediction signal that costs nothing at inference. Same size class as Byrne-VE. ## Related - [Byrne-VE](https://huggingface.co/Quazim0t0/Byrne-VE) - HRM-only sibling - [Escarda-Docling-126M](https://huggingface.co/Quazim0t0/Escarda-Docling-126M) - document VLM on this encoder - [Escarda-86M-Base](https://huggingface.co/Quazim0t0/Escarda-86M-Base)

--- license: apache-2.0 tags: - vision-encoder - jepa - hrm - dinov2-distillation --- # Escarda-VE **Escarda-VE** is the vision encoder of the **Escarda** model family (~39.6M params, 448px / patch16 / 784 tokens). It is the **Byrne-VE** architecture **plus a JEPA head added alongside the HRM refinement block** — Escarda's defining family trait. - ViT-style: RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM refinement. - **JEPA head** (auxiliary): from a patch token it predicts the raster-order neighbour patch's own representation (stop-gradient target, 1−cosine loss). **Zero inference cost.** - Trained by distillation from a frozen **DINOv2-base** teacher (50k steps, CLS + patch), then 20k steps of DINO-style **self-distillation** (EMA teacher, cosine prototypes, no collapse). ## Escarda-VE vs Byrne-VE (DINOv2 teacher-alignment, n=1024 held-out) | | Byrne-VE | Escarda-VE | |---|---|---| | Params | 39.34M | 39.60M (+JEPA head) | | CLS cosine | **0.776** | 0.771 | | PATCH cosine | **0.600** | 0.584 | | JEPA self-consistency | — | 0.040 | **Read:** Escarda trades ~1–3% teacher-alignment (the JEPA objective diverts a little capacity away from pure supervised mimicry) in exchange for a self-supervised spatial neighbour-prediction signal that costs nothing at inference. Same size class as Byrne-VE. ## Related - [Byrne-VE](https://huggingface.co/Quazim0t0/Byrne-VE) — HRM-only sibling - [Escarda-Docling-126M](https://huggingface.co/Quazim0t0/Escarda-Docling-126M) — document VLM built on this encoder - [Escarda-86M-Base](https://huggingface.co/Quazim0t0/Escarda-86M-Base)

Source context: 0 downloads · 1 likes · Repo Quazim0t0/Escarda-VE