Beforeat Food Nutrition Vision LoRA

Intended Use

This adapter is intended for research and prototype food recognition workflows where an image of a prepared dish is provided and the model returns a concise dish-name prediction.

Example prompt:

Identify the dish in this image. Reply with only the exact Food-101 class name.

For a production app, the recommended v1 deployment path is backend inference: the mobile app uploads a food image to a server, the server runs the base Qwen3-VL model with this LoRA adapter, and the app receives the predicted dish name.

Out-of-Scope Use

This checkpoint should not be treated as:

• A complete nutrition estimator
• A medical, dietary, or allergy safety tool
• A reliable portion-size estimator
• A production-grade model for all cuisines, restaurant conditions, or user-generated food photos

The adapter currently recognizes dish names better than it estimates ingredients, calories, macros, or micronutrients.

Training Data

This adapter was trained on Food-101 examples formatted as image-and-text instruction data. Food-101 is useful for dish-name research, but this checkpoint should be treated as an R&D prototype and downstream users should review the Food-101 dataset terms before any commercial or public deployment.

No Food-101 images are included in this adapter repository.

Training Procedure

The adapter was trained locally with PyTorch and PEFT LoRA on Apple Silicon using MPS acceleration.

LoRA configuration:

• Rank: 8
• Alpha: 16
• Dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: about 16.5M

Training progression:

• 1000-step Food-101 adapter
• 1500-step broad Food-101 continuation
• 2000-step broad Food-101 continuation
• 2500-step mixed continuation with broad Food-101 coverage plus extra hard-class examples

Evaluation

Evaluation was run on a fixed 500-image Food-101 sample.

The 2500-step mixed adapter is the current best strict-label checkpoint. The 2000-step broad adapter remains a close fallback because it has slightly higher alias/near scoring.

Common remaining errors include visually similar food categories such as:

• tuna tartare vs beef tartare
• chocolate cake vs chocolate mousse
• donuts vs beignets
• steak, filet mignon, and prime rib
• bread pudding, panna cotta, and dessert-adjacent classes

Loading Example

import torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base_model_id = "Qwen/Qwen3-VL-4B-Instruct"
adapter_id = "YOUR_USERNAME/beforeat-food-nutrition-vision-lora"

processor = AutoProcessor.from_pretrained(
    base_model_id,
    trust_remote_code=True,
)

model = Qwen3VLForConditionalGeneration.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

llama.cpp GGUF Adapter

This repository also includes a standalone llama.cpp-compatible GGUF LoRA adapter for offline runtime loading:

• GGUF adapter filename: beforeat-food-nutrition-vision-lora-f16.gguf
• File size: 33065184 bytes
• SHA-256: 6fa0950b1bc4df4c1018b60ff4e0d824789bbc52a3764d1d38078861a0c2d862
• Format: F16 GGUF LoRA adapter
• Recommended adapter scale: 1.0
• llama.cpp commit used: 00fa7cb284cbf133fc426733bd64238a3588a33e

Required runtime files:

• Base GGUF: Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf
• Vision projector: mmproj-F16.gguf
• Base GGUF source: unsloth/Qwen3-VL-4B-Instruct-GGUF

Conversion command:

PYTHONPATH=tools/llama.cpp \
python tools/llama.cpp/convert_lora_to_gguf.py \
  adapters/qwen3vl4b_food101_peft_2500_mixed_hard20 \
  --base models/Qwen3-VL-4B-Instruct-bf16-remapped \
  --outfile adapters/qwen3vl4b_food101_peft_2500_mixed_hard20/beforeat-food-nutrition-vision-lora-f16.gguf \
  --outtype f16 \
  --verbose

Validation/runtime command:

tools/llama.cpp/build/bin/llama-mtmd-cli \
  -m gguf_base/unsloth-Qwen3-VL-4B-Instruct-GGUF/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj gguf_base/unsloth-Qwen3-VL-4B-Instruct-GGUF/mmproj-F16.gguf \
  --image food-101/images/prime_rib/2421701.jpg \
  --lora-scaled adapters/qwen3vl4b_food101_peft_2500_mixed_hard20/beforeat-food-nutrition-vision-lora-f16.gguf:1.0 \
  -p "Identify the dish in this image. Reply with only the exact Food-101 class name." \
  --jinja \
  --temp 0 \
  -n 16 \
  -c 4096 \
  --image-min-tokens 1024 \
  --no-warmup \
  --no-perf \
  --log-verbosity 1

Runtime validation confirmed that llama.cpp accepts the adapter with the Unsloth UD-Q4_K_XL base and mmproj-F16.gguf, loading all 504 LoRA tensors with no missing, incompatible, or unexpected tensor errors.

Small hard-class validation slice:

On this intentionally hard 10-image slice:

• PyTorch PEFT: 6/10 strict, 6/10 alias/near
• Base GGUF without LoRA: 0/10 strict, 0/10 alias/near
• Base GGUF with GGUF LoRA: 7/10 strict, 8/10 alias/near

Limitations

• The model was evaluated on Food-101-style images, not yet on a broad real-world Beforeat app photo set.
• It may produce close but non-canonical dish names.
• It may confuse visually similar dishes.
• It should not be used as the sole source for dietary, medical, or nutrition decisions.
• Broader ingredient and nutrition recognition requires additional datasets and evaluation.
• The GGUF LoRA adapter was validated with Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf and mmproj-F16.gguf; other base quantizations should be tested before app release.

Recommended Next Steps

• Test this adapter against real Beforeat-style food photos.
• Compare against the 2000-step broad adapter on real app images.
• Add ingredient and nutrition-specific training/evaluation data before claiming nutrition estimation capability.
• Consider a smaller distilled/mobile model for eventual on-device inference.

Framework Versions

• PEFT 0.19.1
• Transformers 4.57.0
• PyTorch nightly used locally for Apple MPS compatibility