SAVANT Scene Description LoRA

--- library_name: peft base_model: Qwen/Qwen2.5-VL-7B-Instruct tags: - anomaly-detection - autonomous-driving - lora - qwen2.5-vl - vision-language-model - scene-description license: apache-2.0 pipeline_tag: image-to-text --- # SAVANT Scene Description Model (LoRA Adapter) This repository contains the LoRA adapter for the scene description model (Phase 1) described in the SAVANT paper, currently under peer review. **This repository is provided for peer-review purposes only.** After the review process, the model will be made publicly available through the authors' main account. ## Model Description LoRA adapter for **Qwen/Qwen2.5-VL-7B-Instruct**, fine-tuned to generate structured scene descriptions from driving scene images. This is **Phase 1** of the SAVANT two-phase pipeline. Given a front-camera image, the model produces a structured JSON description across four semantic layers: - **Street layer:** geometry, topology, surface condition, lane markings - **Infrastructure layer:** traffic lights, signs, cones, barriers, construction sites - **Movable objects layer:** vehicles, pedestrians, other dynamic objects - **Environmental layer:** weather, visibility, lighting conditions ## Training Details - **Base model:** Qwen/Qwen2.5-VL-7B-Instruct - **Method:** LoRA (Low-Rank Adaptation) - **Dataset:** 4,260 samples with structured scene descriptions - **Epochs:** 3 - **Learning rate:** 1e-4 (cosine schedule) - **Precision:** bfloat16 with Flash Attention 2 ### LoRA Configuration | Parameter | Value | |-----------|-------| | Rank (r) | 16 | | Alpha | 32 | | Dropout | 0.05 | | Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, fc1, fc2, qkv, mlp.0, mlp.2 | ## Usage ```python from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor from peft import PeftModel import torch base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-VL-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto" ) model = PeftModel.from_pretrained(base_model, "u94fmn391j/SAVANT-scene-description-lora") processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct") ``` ## Limitations - Trained on the CODA dataset; generalization to other driving domains not evaluated - Single-frame analysis only (no temporal context)