Ukr Council Extractor Gemma 2 2b

decision_no · date (ISO YYYY-MM-DD) · title · signatory · council_name

Results

Evaluated on 10 councils held out entirely from training (none of their documents were seen), so the number measures generalization to unseen councils, not memorization. The metric is strict macro all-fields-exact (afe): the fraction of documents where all five fields match the gold exactly (text fields lower-cased + punctuation-folded, then exact; decision_no space-insensitive exact; date ISO exact), averaged per council so large councils can't dominate.

• Across-seed quality (3 training seeds, full 41-council pool): afe 0.862 ± 0.038. The single-number table above is this published seed-1 checkpoint.
• JSON validity: 100.0%.
• Distillation generalizes: the base model scores 0.146 zero-shot with the identical prompt; the adapter lifts it to 0.911 on councils it never trained on.

Gemma-2 note: load the base with attn_implementation="eager" — the default SDPA path with logit soft-capping silently degrades this model to near-zero accuracy.

Usage

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-2-2b-it"
ADAPTER = "oshyshatskyi/ukr-council-extractor-gemma-2-2b"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="eager")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

SYSTEM = "Ти витягуєш дані з шапки та підпису РІШЕННЯ місцевої ради України. Поверни СТРОГО JSON з рівно такими полями: decision_no, date, title, signatory, council_name. Використовуй null, якщо поле відсутнє у тексті.\n- decision_no: номер рішення БЕЗ префікса «No» і без слова «No». Напр. «No 4227» → «4227»; «No62-2275/2025» → «62-2275/2025». Зберігай складений формат (дефіси, косі риски, роки). null, якщо у шапці немає номера.\n- date: дата ухвалення у форматі ISO РРРР-ММ-ДД. Перетвори з української: «23 грудня 2025 року» → «2025-12-23»; «24.12.2025р.» → «2025-12-24».\n- title: заголовок «Про ...» (повністю, без номера).\n- signatory: ПІБ особи, що підписала (зазвичай голова/секретар ради), зазвичай в кінці. null, якщо не видно.\n- council_name: повна назва ради з шапки."

def head_tail(text, head=2200, tail=800):
    # feed the model the same header+signature window it trained on; elide the middle
    return text if len(text) <= head + tail else text[:head] + "\n...\n" + text[-tail:]

def extract(document_text):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": head_tail(document_text)}]
    prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    ids = tok(prompt, return_tensors="pt").to(model.device)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False)
    return json.loads(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

# print(extract(open("some_decision.txt", encoding="utf-8").read()))

• The base model google/gemma-2-2b-it is gated: accept Google's Gemma license on its Hub page and authenticate (huggingface-cli login) before loading.

Training

• Method: QLoRA — 4-bit NF4 base (double-quant, bf16 compute) + LoRA (r=16, α=32, dropout=0.05) on all attention and MLP projections.
• Optimizer: lr 2e-4, 3 epochs, input window 2560 tokens.
• Data: ~1.6k teacher-labelled decision spans pooled from 41 training councils (10 councils fully held out for evaluation). One prompt for teacher, base, and student, so the comparison is fair.
• Hardware: single H100 (Modal); a training run costs well under $2.

Limitations

• Ukrainian, decision headers only. Built for the рішення header/signature block; not for appendices, voting protocols, or other document types, and not for other languages.
• Strict-exact metric. The ~0.88 afe is conjunctive over five fields; individual fields are 0.95–0.99. The remaining errors are dominated by signatory and by verbatim title/council_name phrasing.
• Prompt-coupled. Use the exact system prompt and head+tail window above; other prompts degrade it.
• Provenance: part of a cross-council generalization study (frozen 51-council dataset). Each field's score above is read from this adapter's own eval artifact.

License

Adapter weights: gemma. You must also comply with the base model's license.