ConvNeXt Food CLF 75

GitHub Repo License: Apache 2.0

F1 Macro mAP

HF Space

ConvNeXt-Food-CLF-75

Multi‐label food ingredient classifier (75 classes) based on ConvNeXt‐Tiny with CBAM attention and GeM pooling.
Trained on a heavily cleaned subset of MM‐Food‐100K.
The model serves as a semantic guide for a downstream segmentation pipeline, but can also be used standalone for ingredient recognition.

📊 Key Metrics: Macro F1 = 0.6546 | mAP = 0.7142 | Validation Loss = 0.0489

💡 Full source code, training scripts, and detailed dataset preparation are available in the GitHub repository.
This model card focuses on how to use the model. For deeper technical details, see the links throughout or visit the GitHub repo.

---

Model Description

ConvNeXt-Food-CLF-75 takes an RGB food image and predicts which of 75 ingredients it contains.
Each prediction is a multi‐label probability – a dish can contain multiple ingredients simultaneously (e.g., “rice” and “chicken”).

• Input: RGB image (any size, automatically resized to 640×640 and normalized with ImageNet statistics)
• Output:
  • A list of ingredient names (default threshold 0.5)
  • Raw probability scores for each of the 75 classes
  • Optional: feature embedding from the penultimate layer (for transfer learning)

Supported Classes (75 ingredients)

almond	cheese butter	grape	onion	sauce
apple	chicken duck	ice cream	orange	sausage
asparagus	chocolate	juice	pasta	seaweed
avocado	cilantro mint	kiwi	peach	shellfish
banana	coffee	lamb	peanut	shrimp
beans	corn	lemon	pear	soup
biscuit	crab	lettuce	peas	spring onion
bread	cucumber	mango	pepper	sprouts
broccoli	date	meat	pineapple	steak
cabbage	egg	melon	pizza	strawberry
cake	eggplant	milk	pork	tea
carrot	fish	mushroom	potato	tofu
cashew	french fries	noodle	pumpkin	tomato
cauliflower	garlic	okra	rice	watermelon
celery stick	ginger	olives	salad	wonton dumplings

---

Intended Uses

• Food ingredient classification into 75 categories independently or for downstream systems (calorie estimation, dietary logging, recipe retrieval).
• Feature extraction: the embedding before the final layer can be fed into a segmentation network to condition mask prediction.
• Research on multi‐label classification with long‐tail distributions and noisy training data.

---

Bias, Risks, and Limitations

• Dataset bias: Trained on a cleaned subset of MM‐Food‐100K. It may underrepresent cuisines not well covered in that dataset.
• Class imbalance: The smallest classes contain as few as ~100 training images; performance may be lower for those ingredients.
• Ambiguity: Visually similar dishes (e.g., “apple” vs. “peach”) may be confused.
• Hallucinations: The model can occasionally predict ingredients that are not visible, especially when cues are subtle or missing.

Recommendations

• Always use the provided preprocessing pipeline (resize to 640×640, ImageNet normalization).
• Consider using Test‐Time Augmentation (TTA) for critical applications.
• Validate predictions against your own data distribution if you plan to deploy the model.

Out‐of‐Scope Use

• Not intended for medical diagnosis or dietary prescription.
• Not a zero‐shot classifier – it only covers the 75 classes listed above.

---

Training Procedure

Full training logs, hyperparameters, and stage‐by‐stage breakdown are in the GitHub model documentation.

Model Architecture

Component	Details
Backbone	ConvNeXt‐Tiny (pretrained on ImageNet‐1k)
Attention	CBAM (Channel + Spatial)
Pooling	GeM (Generalized Mean) with learnable p
Head	Linear(1024 → 512) → GELU → Dropout(0.3) → Linear(512 → 75)
Input	RGB image resized to 640×640
Output	Multi‐label logits (binary relevance per ingredient)

Training Configuration

Parameter	Value
Loss	BCEWithLogitsLoss (class weights: background=1, others=30)
Optimizer	AdamW (head LR=1e‐4, backbone LR=5e‐6)
Scheduler	OneCycleLR (early stages), CosineAnnealingWarmRestarts (later)
Weight decay	1e‐4
Batch size	10
Input size	640×640
MixUp	p=0.5, α=0.2
Mixed precision	AMP (torch.amp)

Training Stages

1. Feature Extraction (28 epochs): Only the classification head is trained, backbone frozen.
2. Discriminative Fine‐Tuning (8 epochs): Head + stages 2‐3 of backbone unfrozen, lower LR for backbone.
3. Staged Unfreezing (1 epoch): Head is trained again to adapt to newly unfrozen features.
4. Final Feature Extraction (5 epochs): Only the classification head is trained at a low LR (5e‐5).

Stage‐by‐Stage Results

Stage	Epoch	F1 (macro)	mAP	Val Loss
Feature Extraction	28	0.3533	0.5067	0.0637
Discriminative Fine‐Tuning	36	0.6121	0.7017	0.0469
Staged Unfreezing	37	0.6426	0.7097	0.0479
Final Feature Extraction	40	0.6546	0.7142	0.0489

---

Evaluation Results (Final Model)

Metric	Value
Macro F1	0.6546
mAP	0.7142
Validation loss	0.0489

Validation set: 2,000 fixed images, split before training.

---

Hardware and Training Time

Component	Detail
GPU	NVIDIA RTX 2060 (6 GB VRAM)
CPU	Intel Xeon E5‐2660 v3
OS	CachyOS (Arch‐based)
Python	3.10+
PyTorch	2.5
timm	1.0.9
Training time	≈1 hour per epoch
Batch size	10
Mixed precision	AMP

Despite the limited 6 GB VRAM, the model trains comfortably with a batch size of 10 at 640×640 resolution.

---

Dataset

The model was trained on a cleaned subset of MM‐Food‐100K, a large‐scale food dataset with noisy 4,000+ ingredient labels.
A rigorous multi‐step cleaning pipeline reduced the original chaos to 75 balanced classes:

• Manually curated target class list (103 initial classes).
• Extracted all matching ingredient names from the dataset.
• Merged semantically identical ingredients (e.g., “enoki mushrooms” + “shiitake mushrooms” → “mushroom”).
• Removed irrelevant matches (e.g., “cheesecake” for the “cheese” class).
• Filtered out classes with fewer than 100 images.
• Created multi‐hot label vectors and serialized the final dataset.

Result: ~88k images (86.3k train, 2k validation), 75 classes.

🔗 The complete cleaning pipeline with all scripts and documentation is here:
dataset_preparation/ on GitHub

---

How to Use

👉 Try it live on Spaces - no installation required!

Quick inference using the provided script (recommended)

The repository includes a self‐contained inference.py script. First, clone this HF repo or download the files, then:

Linux / MacOS:

git clone https://huggingface.co/Alas-V/ConvNeXt-Food-CLF-75
cd ConvNeXt-Food-CLF-75
python -m venv venv
source venv/bin/activate
pip install -r requirements-inference.txt
python -m model.inference --image path/to/meal.jpg --checkpoint pytorch_model.bin --mode text...