--- license: apache-2.0 language: - en - ro library_name: mlx pipeline_tag: text-classification tags: - scam-detection - fraud-detection - smishing - phishing - on-device - mlx - gguf - qwen3 - safety - romanian base_model: Qwen/Qwen3-0.6B datasets: - flowxai/scamguardbench model-index: - name: scam-guard-qwen06b results: - task: type: text-classification name: Scam verdict classification (ScamGuardBench v0.2, 120-item slice) dataset: name: ScamGuardBench v0.2 type: flowxai/scamguardbench metrics: - type: f1 name: verdict micro-F1 value: 0.958 - type: f1 name: verdict macro-F1 value: 0.926 - type: f1 name: tactic macro-F1 value: 0.951 - type: recall name: evidence pass rate value: 0.980 - type: false_positive_rate name: legit-confusable FP-rate (scam_likely on legit) value: 0.000 - task: type: text-classification name: Out-of-distribution fresh CERT-pattern messages (20-item hand-authored set) dataset: name: ScamGuardBench v0.2 type: flowxai/scamguardbench metrics: - type: accuracy name: verdict accuracy (correct / 20) value: 0.90 - type: f1 name: verdict macro-F1 (OOD) value: 0.614 - type: false_positive_rate name: OOD legit false-alarm rate value: 0.000 --- ## Inference contract Running this model correctly requires its **frozen inference contract** — the exact system prompt, output JSON schema, user-turn format, and constrained-decode spec it was trained against. See [`inference_contract/`](./inference_contract): - [`INFERENCE.md`](./inference_contract/INFERENCE.md) — wiring guide: system prompt, user turn `[channel: ]\n ` (tag `sms`/`email`/`chat`), **constrained JSON decoding** (required — pins the enums), and the verbatim-evidence check. - [`prompt_scamguard_sys_v1.txt`](./inference_contract/prompt_scamguard_sys_v1.txt) — the system prompt, verbatim. - [`schema_scamguard_v1.json`](./inference_contract/schema_scamguard_v1.json) — output JSON Schema for constrained decoding. Prompt version `scamguard_sys_v1`. Do not edit the prompt/schema; the weights are trained against them. -qwen06b, its sibling flowxai/ -qwen17b. This is the 0.6B (on-device) card. The 1.7B (quality) card is a sibling repo: release/README_qwen17b.md -> flowxai/scam-guard-qwen17b. The fine-tuned synthetic-bench metrics are PROVISIONAL-on-synthetic-bench; the out-of-distribution fresh-message results are measured and included. All fine-tuned numbers on this card are the FINAL CUDA 3-epoch run (superseding the MLX first pass). --> # scam-guard 0.6B (working name) — the on-device pick **An on-device scam & fraud message detector for SMS, email, and chat text (English + Romanian).** This is the **0.6B** model — the **smallest and fastest** of the two scam-guard sizes, and the **on-device target** (~0.4 GB at int4). For higher out-of-distribution accuracy at a larger footprint, see the sibling **[1.7B quality pick](https://huggingface.co/flowxai/scam-guard-qwen17b)**. Given one message, scam-guard returns a **3-level verdict**, the **manipulation tactics** it found (each with a verbatim evidence span quoted from the message), a **calm plain-language explanation**, and a **recommended safe action** from a fixed list. It is built for everyday people — explicitly including **elderly and non-technical users**, who are the most targeted. The product insight that shapes everything: **the `suspicious` middle level exists to be honest about uncertainty rather than force a binary.** A consumer safety tool that must answer "scam or not" will either cry wolf or wave real scams through; a third honest verdict — "this might be fine, verify through your own channel first" — lets the model say *I'm not sure* instead of guessing. Relatedly, scam-guard **never emits a probability**: an uncalibrated confidence number on a consumer safety tool is worse than none, so we give you honest per-class behaviour instead (see [Calibration](#calibration--we-dont-give-you-a-probability)). - **`flowxai/scam-guard-qwen06b`** (this card, 0.6B) — smallest and fastest; the on-device target. - **`flowxai/scam-guard-qwen17b`** (1.7B, sibling) — the more robust choice out-of-distribution (see [Size decision](#size-decision)). Both are LoRA-fine-tuned from Apache-2.0 Qwen3 base models. On-device formats: **GGUF** (llama.cpp, int8 `Q8_0` + int4 `Q4_K_M`) and **MLX-quantized** (int4 + int8; mlx-swift runs these on iOS too). --- ## How do I use it? Three copy-pasteable ways to turn a message into a verdict. All run **fully on-device** — no network at inference, ever. Real example input (a fresh Romanian courier-fee smishing message): ``` Coletul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata Livrarea se anuleaza in 48h. ``` ### (a) llama.cpp / GGUF Download a GGUF (int8 `Q8_0` recommended) and run the message through it. The model emits a single strict JSON object. **`llama-cli` (CPU-only, `-ngl 0`):** ```bash llama-cli -m scam-guard-qwen06b-Q8_0.gguf -ngl 0 --temp 0 -no-cnv \ -p "$(cat [channel: sms] Coletul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata Livrarea se anuleaza in 48h. EOF )" ``` **`llama-cpp-python`:** ```python from llama_cpp import Llama from scamguard.schema import SYSTEM_PROMPT, ScamGuardOutput # the fixed task prompt + schema llm = Llama(model_path="scam-guard-qwen06b-Q8_0.gguf", n_gpu_layers=0) msg = ("Coletul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati " "taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata " "Livrarea se anuleaza in 48h.") out = llm.create_chat_completion( messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"[channel: sms]\n{msg}"}, ], temperature=0.0, ) raw = out["choices"][0]["message"]["content"] verdict = ScamGuardOutput.model_validate_json(raw) # strict, extra="forbid" ``` ### (b) MLX (Apple Silicon) Off the MLX-quantized weights (int4/int8): ```bash mlx_lm.generate --model scam-guard-qwen06b-mlx-int4 --temp 0 \ --prompt "$(printf '[channel: sms]\nColetul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata Livrarea se anuleaza in 48h.')" ``` ```python from mlx_lm import load, generate from scamguard.schema import SYSTEM_PROMPT model, tok = load("scam-guard-qwen06b-mlx-int4") msg = ("Coletul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati " "taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata " "Livrarea se anuleaza in 48h.") prompt = tok.apply_chat_template( [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"[channel: sms]\n{msg}"}], add_generation_prompt=True, enable_thinking=False, ) raw = generate(model, tok, prompt=prompt, max_tokens=256, verbose=False) ``` Either backend returns the **same strict JSON** for this message: ```json { "verdict": "scam_likely", "tactics": [ { "tactic": "subscription_trap", "evidence": "achitati taxa vamala de 3,20 lei", "explanation": "It asks you to pay a small fee to release a parcel." }, { "tactic": "urgency_pressure", "evidence": "se anuleaza in 48h", "explanation": "It invents a 48-hour deadline to rush you." } ], "explanation": "This looks like a scam because it uses a fake fee to prompt a payment and pressures you with an artificial deadline; do not act on it, and check with the real organisation through a channel you already trust.", "recommended_action": "verify_via_official_app_or_site" } ``` ### (c) The demo verdict card (`demo/check.py`) The reference demo renders that JSON as a card a family member can read. It hard-enforces the no-network privacy promise: ```bash echo "Coletul dumneavoastra nu a putut fi livrat. Pentru reprogramare achitati taxa vamala de 3,20 lei aici: http://colet-reprogramare.example.net/plata Livrarea se anuleaza in 48h." | python demo/check.py ``` Rendered output (the actual card, from `reports/ood_fresh_demo.md`): ``` ==================================================================== scam-guard — message safety check ==================================================================== [!] VERDICT: Likely a scam This message shows clear signs of a scam. You do not need to do anything it asks. Take your time — real organisations are fine with you checking first. -------------------------------------------------------------------- What we noticed: - Sending a fake renewal or invoice to make you call or click seen in: "achitati taxa vamala de 3,20 lei" - Rushing you with a deadline or threat seen in: "se anuleaza in 48h" -------------------------------------------------------------------- What to do: Check directly using the company's official app or website that you open yourself — not the link here. -------------------------------------------------------------------- In plain words: This looks like a scam because it uses a fake renewal invoice to prompt a call and it pressures you with an artificial deadline; do not act on it, and check with the real organisation through a channel you already trust. ==================================================================== scam-guard is a helper, not a guarantee. When in doubt, verify through a channel you already trust. It never opens links. ==================================================================== ``` `demo/check.py` defaults to the 0.6B MLX model (this one); `--backend gguf` and `--model ` switch weights/backend, and `--size 1.7b` selects the sibling. `--channel {sms,email,chat}` sets the channel tag. --- ## How it works ```mermaid flowchart LR SMS --> SG["scam-guard 0.6B"] Email --> SG Chat --> SG SG --> V[verdict] SG --> T[tactics] SG --> E[evidence] SG --> X[explanation] SG --> A[action] ``` Compact ASCII flow — three channels in, one local model, five fields out: ``` SMS \ Email ---> scam-guard 0.6B Chat / | + verdict + tactics + evidence + explanation + action ``` Under the hood, `scam-guard` is: `tokenizer → Qwen3-0.6B (LoRA fine-tuned) → constrained JSON decode → evidence verifier (verbatim-substring kill-switch: drops fabricated spans)`. **The evidence kill-switch is the safety-critical stage.** Every tactic must cite a span that is a **verbatim substring** of the input message (whitespace-normalized only — no case/diacritic folding). A tactic whose evidence is not found verbatim is **dropped and counted** as fabricated, so the model can never hallucinate a quote to justify a warning. --- ## Output schema A single strict JSON object (`extra="forbid"`, frozen — a spurious field like `confidence` is rejected): - `verdict` — `scam_likely` | `suspicious` | `no_indicators` (**never a probability**). - `scam_likely` — a clear scam mechanism is present and driven by tactics. - `suspicious` — the honest middle: signals present but plausibly legitimate, or only weak indicators (urgency alone, a link alone, an authority claim with no ask). "Verify through your own channel first." - `no_indicators` — no scam mechanism; legitimate messages can be urgent and contain links. - `tactics[]` — each `{tactic, evidence, explanation}`, where `tactic` is one of 13 fixed ids and `evidence` is a **verbatim substring** (see the kill-switch above). - `explanation` — one or two calm, actionable sentences. - `recommended_action` — one id from a fixed list of 10 safe actions; the model can never compose free-text advice that points back at the scammer's own channel. The 13 tactics: `urgency_pressure`, `authority_impersonation`, `payment_redirect`, `credential_phishing`, `courier_customs_fee`, `prize_lottery`, `investment_too_good`, `romance_advance_fee`, `family_emergency_impersonation`, `tech_support`, `link_obfuscation`, `refund_overpayment`, `subscription_trap`. The 10 safe actions: `call_bank_official_number`, `do_not_click_link`, `verify_via_official_app_or_site`, `call_family_member_known_number`,...
