Loading
Du hast einen ComfyUI-Workflow? Erstelle direkt im Browser eine verständliche Workflow-Dokumentation.Workflow verständlich machen.Jetzt ausprobieren ↗
research-only
Modellquelle
Quellenbeschreibung
A single multi-task convolutional model for facial behavior analysis, used by
py-feat's Detectorv2. From one face crop
it jointly predicts action units, categorical emotion, valence/arousal,
eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
blendshapes (the v2.7 model; replaces v2.6).
Quellen
1 QuelleVerifiziert 17. Sept.
Modellartefakte
5 Artefakteface_multitask_v2.safetensors
safetensors · 146 MB · SHA-256 076901ef5804…edde · Hugging Face
HerunterladenQuellenauszüge
2 Auszüge
face_multitask_v27.safetensors (safetensors; ModelV2Config JSON in the file metadata)| Task | Output | Notes |
|---|---|---|
| Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 |
| Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger |
| Valence / Arousal | 2 × [−1,1] | tanh |
| Gaze | (yaw, pitch) radians | RAW convention is y-down: yaw+ = subject's right (image-left), pitch+ = looking DOWN. Detectorv2 negates pitch so Fex columns are canonical +up (since py-feat 2.1.1) |
| Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) |
| Head pose | (pitch, yaw, roll, tx, ty, tz) | radians / pixels; RAW pitch+ = down (img2pose teacher frame); Detectorv2 outputs canonical +up (since py-feat 2.1.1) |
| 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
| Blendshapes | 52 coefficients [0,1] |
All gaze splits are identity-disjoint from training (held-out subjects), and EYEDIAP is never trained on by any py-feat model.
| Task | Dataset | Metric | v2.7 | v2.6 | v2.5 |
|---|---|---|---|---|---|
| AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | 0.686 | 0.696 | 0.693 |
| AU | DISFA+ (8-AU subset) | macro-F1 | 0.740 | 0.738 | 0.740 |
| Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | 0.612 / 0.607 | 0.615 / 0.610 | 0.616 / 0.612 |
| Emotion | RAF-DB test | acc / macro-F1 | 0.876 / 0.817 | 0.873 / 0.818 | 0.910 / 0.885 |
| Valence/Arousal | AffectNet val | CCC (V / A) |
Notes: v2.7 = the v2.6 architecture retrained with a rescaled gaze loss, a rebalanced within-gaze data mix, per-source augmentation, 2° gaze bins, and a head-pose label fix. Vs v2.6 it improves out-of-distribution gaze (EYEDIAP −0.7°, Columbia −1.3°), RAF-DB macro-F1 end-to-end (+2.7), AFEW-VA arousal, blendshape fidelity (best of any release), and occlusion robustness, at the cost of ~0.6° on frontal MPIIGaze and 0.01 on the 12-AU set.
Known limitation (v2.6 and v2.7): continuous valence/arousal on
video-frame corpora (Aff-Wild2, AFEW-VA) is substantially below v2.5
(e.g. Aff-Wild2 CCC-V 0.85 → 0.33). AffectNet (still-image) V/A is unaffected.
If frame-wise continuous V/A on video is your primary measure, prefer the v2.5
weights (face_multitask_v2.safetensors, still published in this repo). A fix
is under investigation. Numbers are from the deployed checkpoint (v2.7
stage-3 soup ep05-09), weight-verified against the published .safetensors.
from feat import Detectorv2
detector = Detectorv2(device="cuda")
fex = detector.detect("image.jpg") # returns a py-feat Fex
The model expects a face crop produced by RetinaFace + py-feat's
extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2),
then resized to 224 and ImageNet-normalized. Detectorv2 handles this.
Research / non-commercial use only. Trained on datasets (AffectNet, DISFA+, RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research. The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's terms before any non-research use.
face_multitask_v26.safetensors
safetensors · 157 MB · SHA-256 35a41175e0e4…1baa · Hugging Face
face_multitask_v27.safetensors
safetensors · 159 MB · SHA-256 e3c870e22ba0…c8bd · Hugging Face
Herunterladenface_multitask_v28.safetensors
safetensors · 159 MB · SHA-256 e8bf7fe0a1a3…992e · Hugging Face
Herunterladen--- license: other license_name: research-only license_link: LICENSE library_name: py-feat tags: - facial-expression-analysis - action-units - emotion-recognition - gaze-estimation - face-landmarks - head-pose - blendshapes - multitask pipeline_tag: image-classification --- # face_multitask_v2 A single multi-task convolutional model for facial behavior analysis, used by [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop it jointly predicts **action units, categorical emotion, valence/arousal, eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit blendshapes** (the v2.7 model; replaces v2.6). - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained) - **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads + landmark, pose, and **blendshape** regression heads + the v2.6 **eye-aware gaze head**: RoI-pooled eye features (localized by the predicted mesh), conditioned on predicted head pose (6D), with an L2CS-style binned prediction over the full ±180° range (v2.7: 2° bins) - **Params:** ~42M · **Input:** 224×224 RGB (from a 256×256 face crop) - **File:** `face_multitask_v27.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata) - **Weights:** soup (equal average) of five consecutive fine-tuning checkpoints ## Outputs | Task | Output | Notes | |---|---|---| | Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 | | Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger | | Valence / Arousal | 2 × [−1,1] | tanh | | Gaze | (yaw, pitch) radians | RAW convention is y-down: yaw+ = subject's right (image-left), pitch+ = looking DOWN. `Detectorv2` negates pitch so Fex columns are canonical +up (since py-feat 2.1.1) | | Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) | | Head pose | (pitch, yaw, roll, tx, ty, tz) | radians / pixels; RAW pitch+ = down (img2pose teacher frame); `Detectorv2` outputs canonical +up (since py-feat 2.1.1) | | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh | | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) | ## Benchmarks (held-out, file-verified — v2.7 deployed checkpoint) All gaze splits are identity-disjoint from training (held-out...
Source context: 0 downloads · 1 likes · Pipeline image-classification · Library py-feat · Repo py-feat/face_multitask_v2
| MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, ...) |
| 0.773 / 0.647 |
| 0.775 / 0.653 |
| 0.780 / 0.646 |
| Valence/Arousal | AFEW-VA | CCC (V / A) | 0.711 / 0.480 | 0.718 / 0.411 | 0.833 / 0.863 |
| Valence/Arousal | Aff-Wild2 val | CCC (V / A) | 0.331 / 0.418 | 0.397 / 0.458 | 0.852 / 0.799 |
| Gaze | ETH-XGaze (held-out subjects) | mean angular err | 5.1° | 5.0° | 43.2° |
| Gaze | EYEDIAP (never-train, 15.2K frames) | mean angular err | 12.6° | 13.4° | 15.3° |
| Gaze | Gaze360 (held-out split) | mean angular err | 13.0° | 13.0° | 12.9° |
| Gaze | MPIIGaze (leave-subject-out) | mean angular err | 8.0° | 7.4° | 7.0° |
| Gaze | Columbia (held-out subjects) | mean angular err | 4.1° | 5.4° | — (trained) |
| Blendshapes | FacePlace (teacher agreement) | mean active-ch. r | 0.761 | 0.748 | 0.756 |