--- license: apache-2.0 library_name: sentence-transformers pipeline_tag: sentence-similarity language: - es base_model: - BSC-LT/MrBERT-es tags: - legal - administrative - spanish - bi-encoder - sentence-transformers - embeddings - retrieval --- # ALIA MrBERT Spanish Legal and Administrative Embeddings Model This repository contains **ALIA MrBERT Spanish Legal and Administrative Embeddings**, a Spanish legal domain bi-encoder model for semantic similarity and information retrieval tasks. It is built upon [MrBERT-es](https://huggingface.co/BSC-LT/MrBERT-es), a bilingual (Spanish–English) foundational language model based on the ModernBERT architecture, and fine-tuned on domain-specific legal and administrative data using a Curriculum Learning strategy. > [!WARNING] > **DISCLAIMER:** This model is a domain-specific proof-of-concept designed to demonstrate retrieval capabilities in the Spanish legal and administrative domain. > While optimized for this domain, results should be verified against official legal sources. The model may fail in out-of-domain or adversarial inputs. --- ## Model Details ### Model Lineage ``` ModernBERT (architecture) ↓ MrBERT-es (BSC-LT) Bilingual ES/EN encoder 150M parameters ↓ ALIA-MrBERT-es-legal-administrative-embeddings (SINAI) Legal domain fine-tuning Curriculum Learning + Hard Negatives ``` ### Key Features - 🔍 **Domain**: Spanish legal and administrative texts - 📐 **Architecture**: ModernBERT with Mean Pooling (bi-encoder) - 📏 **Long context**: up to 8,192 tokens - 🎓 **Training strategy**: Curriculum Learning (easy → medium → hard) - ⚙️ **Negative mining**: Positive-Aware Hard Negative Mining (NVIDIA approach) ### Architecture This model uses the same base architecture as [MrBERT-es](https://huggingface.co/BSC-LT/MrBERT-es), extended with a Mean Pooling layer for sentence-level embeddings: | | | |------------------------------|:--------------| | **Base Architecture** | ModernBERT | | **Total Parameters** | ~150M | | **Hidden size** | 768 | | **Intermediate size** | 1,152 | | **Attention heads** | 12 | | **Hidden layers** | 22 | | **Context length** | 8,192 tokens | | **Vocabulary size** | 51,200 | | **Precision** | bfloat16 | | **Positional encoding** | RoPE | | **Activation function** | GeLU | | **Attention type** | Mixed (global every 3 layers + sliding window) | | **Pooling strategy** | Mean Pooling | --- ## Training ### Training Strategy: Curriculum Learning The model was fine-tuned using a two-phase **Curriculum Learning** strategy and progressively increasing the difficulty of training examples thanks to [SINAI/ALIA-es-legal-administrative-triplets/train](https://huggingface.co/datasets/SINAI/ALIA-es-legal-administrative-triplets): | Phase | Epochs | Negative Type | Difficulty Progression | |-------|:------:|:-------------:|:----------------------:| | **Phase 1** | 6 | Random negatives | Easy → Medium → Hard | | **Phase 2** | 3 | Hard negatives (mined) | Easy → Medium → Hard | | **Total** | **9** | — | — | **Phase 1 – Contrastive Learning with Random Negatives:** Training uses `{query, relevant_doc, [irrelevant_docs]}` with in-batch negatives. Examples are sorted by difficulty across 3 sub-phases (2 epochs each). **Phase 2 – Advanced Refinement with Hard Negatives:** Refinement using mined hard negatives with **Positive-Aware Mining** (NVIDIA approach) to avoid false negatives. A candidate is only considered a negative if: ``` score **Note:** Each evaluation subset is named following the pattern `{dataset}_queries{N}_contexts{M}`, where `N` is the number of queries evaluated against `M` contexts taken from the datasets. --- ## Limitations and Biases ### Known Limitations - **Domain specificity**: The model is optimized for Spanish legal and administrative texts. Performance may degrade significantly on general-domain or other specialized texts. - **Language**: Although MrBERT-es supports Spanish and English, this fine-tuned model focuses on Spanish legal content. - **Legal accuracy**: Semantic similarity does not guarantee legal correctness. Retrieved documents should always be verified by qualified professionals. - **Context length**: Despite supporting up to 8,192 tokens, very long documents may require chunking strategies for optimal retrieval performance. ### Biases - The model may reflect biases present in the Spanish legal corpus used for training. - It may underperform on legal texts from Latin American jurisdictions, as training focused on Spanish national legislation and administration. --- ## Additional Information ### License [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0) ### Citation If you use this model in your research, please cite: ```bibtex @misc{ALIA-MrBERT-es-legal-administrative-embeddings, title = {ALIA MrBERT Spanish Legal and Administrative Embeddings Model}, author = {SINAI Research Group, Universidad de Jaén}, year = {2026}, publisher = {HuggingFace}, howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-legal-administrative-embeddings}} } ``` Please also cite the base model: ```bibtex @misc{tamayo2026mrbertmodernmultilingualencoders, title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation}, author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas}, year={2026}, eprint={2602.21379}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2602.21379}, } ``` ### Funding This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project [ALIA](https://alia.gob.es). ### Acknowledgments This dataset has been generated thanks to [CEATIC](https://www.ujaen.es/centros/ceatic/) ( Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – [UJA](http://www.ujaen.es/) (Universidad de Jaén) which provided the needed computational resources on its clusters. --- **Contact:** [ALIA Project](https://www.alia.gob.es/) - [SINAI Research Group](https://sinai.ujaen.es) - [Universidad de Jaén](https://www.ujaen.es/) **More Information:** [SINAI Research Group](https://sinai.ujaen.es) | [ALIA-UJA Project](https://github.com/sinai-uja/ALIA-UJA)
