Scam Qwen3 VL RsLoRA

--- base_model: Qwen/Qwen3-VL-4B-Instruct library_name: peft pipeline_tag: image-text-to-text language: - en - tl license: cc0-1.0 tags: - base_model:adapter:Qwen/Qwen3-VL-4B-Instruct - lora - rslora - peft - transformers - scam-detection - video-understanding - multimodal - qwen3-vl --- # Scam-Qwen3-VL-RsLoRA A **Rank-Stabilized LoRA (rsLoRA)** adapter for `Qwen/Qwen3-VL-4B-Instruct`, fine-tuned to detect **crypto / financial scams in short-form social video** (YouTube + TikTok), with a focus on Filipino / Philippine scam content. Given a video's frames plus its title and description, the model emits a chain-of-thought ` ` rationale grounded in a policy criteria scheme (C1–C7) and a structured JSON verdict. ## Model Details ### Model Description This is a PEFT/LoRA **adapter only** — it must be loaded on top of the frozen base model `Qwen/Qwen3-VL-4B-Instruct`. The base weights are never modified; only the rsLoRA adapter (~120 MB) is trained and distributed here. The model takes a multimodal prompt (sampled video frames + text metadata) and produces: 1. A ` ... ` block containing the evidence it found and which policy criteria (C1–C7) were hit, then 2. A JSON object: `{"verdict": "Yes"|"No", "confidence": , "category": }`. - **Developed by:** Jules Gregory R. Agustin (HF: [TamAko783](https://huggingface.co/TamAko783)) - **Model type:** Vision-Language (multimodal) classifier / reasoner — rsLoRA adapter - **Task:** Scam vs. not-scam detection over social-media video with rationale generation - **Language(s):** English, Filipino/Tagalog (code-switched) - **License:** CC0-1.0 (matches the training dataset) - **Finetuned from:** [`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) - **Adapter type:** Rank-Stabilized LoRA (`use_rslora=true`), PEFT 0.19.1 ### Model Sources - **Repository:** https://huggingface.co/TamAko783/Scam-Qwen3-VL-RsLoRA - **Base model:** https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct - **Training dataset:** [OptiScam: Multimodal Filipino Scam Video Dataset](https://www.kaggle.com/datasets/julesgregoryagustin/training-filipino-scam-dataset) ## Uses ### Direct Use Screening / triage of short-form crypto and financial-promotion videos to flag likely scams (giveaways, airdrop bait, fake mining/doublers, get-rich-quick lures, gift-card and play-to-earn schemes), together with a human-readable rationale and policy-criteria citations. ### Downstream Use - A first-pass filter in a moderation pipeline, with the ` ` trace surfaced to a human reviewer. - A research baseline for multimodal scam detection and LLM-as-a-Judge reasoning evaluation. ### Out-of-Scope Use - **Not** an automated enforcement / takedown system. Use human-in-the-loop review. - Not validated outside the crypto/financial scam domain or on long-form video. - Not a financial-advice or investment-recommendation tool. - Performance on languages or scam typologies under-represented in the training data is unverified. ## Bias, Risks, and Limitations - **Over-flagging (false positives):** the model sometimes labels legitimate educational/disclaimed content as a scam (e.g., real airdrop explainers, learn-to-earn guides, gift-card demos). - **Under-reading subtle signals (false negatives):** misses domain spoofs (e.g. `m1finance.8bxp97.net`), P2P payment name mismatches, and Telegram/off-platform funnels that piggyback on legitimate branding. - **Occasional hallucination:** rarely invents a domain name not present in the input (e.g. `zoro-loot.com`). - **Decoding artifacts:** in a few cases the ` ` block describes a textbook scam but the final verdict JSON says "No" (self-contradiction). - **Domain/locale bias:** trained on a Philippine-focused corpus; cues may not transfer to other regions. ### Recommendations Always keep a human reviewer in the loop, treat the verdict as advisory, and read the ` ` rationale to catch the failure modes above (over-flagging, contradictory verdicts, fabricated entities). ## How to Get Started with the Model This is an adapter, so you load the **base model first**, then attach the adapter. ```bash pip install -U transformers peft accelerate torchvision qwen-vl-utils ``` ```python import torch from transformers import AutoProcessor, AutoModelForImageTextToText from peft import PeftModel BASE = "Qwen/Qwen3-VL-4B-Instruct" ADAPTER = "TamAko783/Scam-Qwen3-VL-RsLoRA" # 1) Load the frozen base model model = AutoModelForImageTextToText.from_pretrained( BASE, torch_dtype=torch.bfloat16, device_map="auto", ) # 2) Attach the rsLoRA adapter model = PeftModel.from_pretrained(model, ADAPTER) model.eval() # (optional) fuse for faster inference: model = model.merge_and_unload() processor = AutoProcessor.from_pretrained(BASE) # 3) Build a multimodal prompt: sampled video frames + metadata SYSTEM = ("You are an expert scam-detection analyst. Use your native OCR ability " "to analyze the frames provided. Decide if the video is a scam based on " "YouTube's Scams policy and cite policy criteria (C1-C7).") messages = [ {"role": "system", "content": [{"type": "text", "text": SYSTEM}]}, {"role": "user", "content": [ # pass sampled frames (or a {"type": "video", "video": " "} entry) {"type": "image", "image": "frame_00.jpg"}, {"type": "image", "image": "frame_30.jpg"}, {"type": "text", "text": "Title: \nDescription: \n\n" "Is this a scam video? Provide a rationale grounded in Policy Criteria (C1-C7)."}, ]}, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) out = model.generate(**inputs, max_new_tokens=1024, do_sample=False) print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]) ``` **Expected output shape:** ```text Evidence: ... Criteria hits (C1-C7): C2 - promises unbounded free crypto; C4 - fast-money lure ... {"verdict": "Yes", "confidence": 0.95, "category": "Crypto Investment"} ``` > Tips: the model was trained on up to **60 frames per video** at `max_pixels ≈ 200704`. Thinking > (` `) mode benefits from `max_new_tokens=1024`. Greedy decoding (`do_sample=False`) is recommended for reproducible verdicts. ## Training Details ### Training Data - **Dataset:** `julesgregoryagustin/training-filipino-scam-dataset` (CC0-1.0), ShareGPT multimodal format. - ~2,000 YouTube/TikTok videos sampled at **60 frames** each; ~1,600 train / held-out test (n=202 eval items). - Each example: system prompt + video frames + title/description, with a gold `yes`/`no` label and a human-authored rationale ("Why Verdict" + "Criteria Hits", criteria C1–C7). ### Training Procedure #### Training Hyperparameters | Hyperparameter | Value | |---|---| | Adapter | Rank-Stabilized LoRA (`use_rslora=true`) | | LoRA rank `r` | 32 | | LoRA alpha | 32 | | LoRA dropout | 0.1 | | Effective scaling (`α/√r`) | ≈ 5.657 | | Init | Gaussian | | Learning rate | 7e-5 | | Epochs | 4 | | Max frames | 60 | | Max pixels / frame | 200704 (~448×448) | | Precision | bf16 mixed precision | | Base model | frozen | | Final eval loss | 0.884 (epoch 4) | **Target modules:** text attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`), vision attention (`qkv`, `attn.proj`), and the multimodal projector / deepstack mergers (`linear_fc1`, `linear_fc2`). #### Compute Single NVIDIA A100 GPU (Google Colab), PEFT 0.19.1. ## Evaluation All metrics are on the held-out test split (**n = 202**). ### Verdict accuracy | Configuration | Accuracy | Precision | Recall | F1 | TP/FP/TN/FN | |---|---|---|---|---|---| | **rsLoRA — full multimodal, thinking on** (max_new_tokens=1024) | **0.8119** | **0.8387** | **0.7723** | **0.8041** | 78/15/86/23 | | rsLoRA — greedy, no-thinking decode | 0.7772 | 0.8182 | 0.7129 | 0.7619 | 72/16/85/29 | | Base Qwen3-VL-4B — zero-shot (no adapter) | 0.7277 | 0.7556 | 0.6733 | 0.7120 | 68/22/79/33 | The thinking-enabled rsLoRA adapter improves **F1 by ~9 points** over the zero-shot base model (0.804 vs 0.712) and tightens precision (fewer false positives: 15 vs 22). ### Input-modality ablation (greedy, 256 tokens) | Inputs | Accuracy | Precision | Recall | F1 | TP/FP/TN/FN | |---|---|---|---|---|---| | Frames only | 0.7574 | 0.7364 | 0.8020 | 0.7678 | 81/29/72/20 | | Transcript + Title + Description | 0.7228 | 0.7143 | 0.7426 | 0.7282 | 75/30/71/26 | | Transcript only | 0.5446 | 0.5413 | 0.5842 | 0.5619 | 59/50/51/42 | **Takeaway:** the **visual signal dominates**. Frames alone reach F1 ≈ 0.77, while transcript-only collapses toward chance (0.56) — scam cues (on-screen URLs, overlays, wallet QR codes) live in the pixels, which is exactly what the rsLoRA vision-tower adaptation targets. ### Reasoning quality — LLM-as-a-Judge (n = 40 stratified) Reasoning traces were audited with **Claude Opus 4.7** as a cross-family judge under a reference-grounded binary rubric, treating reasoning quality as an inter-rater agreement problem against an outcome-derived rater. | Metric | Value | |---|---| | Percentage agreement (p_o) | 0.9750 (39/40) | | **Cohen's κ** | **0.9500** — *almost perfect* (Landis–Koch) | | BERTScore F1 (mean) | 0.8696 | | ROUGE-L F1 (mean) | 0.2580 | Triangulation with reference-free similarity metrics shows a monotone **TN > TP > FP > FN** gradient (BERTScore F1: 0.9005 > 0.8824 > 0.8531 > 0.8425), and judge-`Correct` items score 0.893 vs 0.849 for judge-`Incorrect` — independent evidence that the model's reasoning is genuinely paraphrase-aligned with the human rationale when its verdict is right (not post-hoc rationalization). ### Observed failure modes | Failure | Severity | Example | |---|---|---| | Over-flagging legitimate content (false positives) | high | misreads disclaimed airdrop/learn-to-earn videos as scams | | Under-reading subtle scam signals (false negatives) | high | misses domain spoofs, P2P name mismatch, Telegram funnels | | Fabricated entities (hallucination) | high | invented `zoro-loot.com` | | Self-contradictory verdict (decoding bug) | medium | ` ` says "scam" but JSON verdict = "No" | ### Policy criteria (C1–C7) The rationales cite a YouTube-Scams-policy-derived criteria scheme. Codes observed in the model's output include: **C2** (promises unbounded/free crypto), **C3** (airdrop-link bait / off-platform redirect), **C4** (fast-money / guaranteed-return lure), **C5** (solicits wallet address / seed phrase), **C6** (get-rich-quick hook), **C7** (brand/channel impersonation). Full legend: **C1** Criminal claim, **C2** Unbounded giveaway, **C3** Off-site redirect, **C4** Fast-money lure, **C5** Harmful link, **C6** Get-rich-quick, **C7** Impersonation. Extended Filipino archetypes: P2E (Play-to-Earn), Task (Telegram / Group Task), and E-Wallet (GCash / Maya phishing). The criteria scheme is adapted from Kulsum et al. (arXiv:2509.23418). ## Technical Specifications ### Model Architecture and Objective Causal-LM (`CAUSAL_LM`) rsLoRA adapter over the Qwen3-VL-4B-Instruct vision-language backbone. Objective: supervised fine-tuning to produce a ` ` rationale + JSON scam verdict from video frames and text metadata. Processor: `Qwen3VLProcessor` (`fps=2`, patch size 16, merge size 2). ### Framework versions - PEFT 0.19.1 - Transformers (Qwen3-VL support) - base_model: `Qwen/Qwen3-VL-4B-Instruct` ## Citation If you use this adapter, please cite the base model and dataset: ```bibtex @misc{agustin2026scamqwen3vl, title = {Scam-Qwen3-VL-RsLoRA: A Rank-Stabilized LoRA Adapter for Multimodal Scam Detection}, author = {Agustin, Jules Gregory R.}, year = {2026}, howpublished = {\url{https://huggingface.co/TamAko783/Scam-Qwen3-VL-RsLoRA}} } ``` **Dataset and criteria inspiration:** > Ummay Kulsum, Aafaq Sabir, Abhinaya S.B., and Anupam Das. "Beyond Metadata: Multimodal, Policy-Aware Detection of YouTube Scam Videos." ICWSM 2026. arXiv:2509.23418. ```bibtex @inproceedings{kulsum2026beyond, title = {Beyond Metadata: Multimodal, Policy-Aware Detection of YouTube Scam Videos}, author...