CalorieCLIP

--- license: mit language: - en tags: - vision - food - nutrition - calorie-estimation - clip - image-classification - health datasets: - nutrition5k metrics: - mae pipeline_tag: image-to-text library_name: open-clip --- # 🍎 CalorieCLIP: Accurate Food Calorie Estimation <p align="center"> <img src="assets/model_comparison.png" width="700" alt="CalorieCLIP vs Other Models"> </p> **CalorieCLIP** is a fine-tuned CLIP model that estimates calories from food images with state-of-the-art accuracy. It outperforms all tested VLMs (including GPT-4o and Claude) while running entirely on-device. ## 🎯 Key Results | Metric | Value | |--------|-------| | **Mean Absolute Error** | **51.4 calories** | | Within 50 calories | 67.6% | | Within 100 calories | 90.5% | | Inference Speed | <50ms on M1 Mac | <p align="center"> <img src="assets/accuracy_breakdown.png" width="500" alt="Accuracy Breakdown"> </p> ## 🍽️ Example Predictions Real predictions from our validation set across multiple datasets: | Image | Food | Dataset | Actual | Predicted | Error | |-------|------|---------|--------|-----------|-------| |  | Hamburger | Food-101 | 558 | 555 | 3 | |  | Ramen | Food-101 | 431 | 437 | 6 | |  | Greek Salad | Food-101 | 144 | 143 | 1 | |  | Sashimi | Food-101 | 156 | 156 | 0 | |  | Cafeteria Meal | Nutrition5k | 88 | 88 | 0 | |  | Cafeteria Meal | Nutrition5k | 138 | 138 | 0 | |  | Cafeteria Meal | Nutrition5k | 330 | 334 | 3 | |  | Cafeteria Meal | Nutrition5k | 214 | 217 | 4 | ## 🚀 Quick Start ### Installation ```bash pip install open-clip-torch torch pillow ``` ### Python Usage ```python # Clone or download this repo first, then: from calorie_clip import CalorieCLIP # Load model from local directory model = CalorieCLIP.from_pretrained(".") # Predict calories calories = model.predict("food_photo.jpg") print(f"Estimated: {calories:.0f} calories") # Batch prediction images = ["breakfast.jpg", "lunch.jpg", "dinner.jpg"] results = model.predict_batch(images) ``` ### Direct Usage (no wrapper) ```python import torch im...

Source context: 26 downloads · 19 likes · Pipeline image-to-text · Library open-clip · Repo jc-builds/CalorieCLIP