1m Paper Embedding Model

--- base_model: allenai/scibert_scivocab_uncased library_name: peft pipeline_tag: feature-extraction tags: - peft - lora - scibert - sentence-transformers - feature-extraction - embeddings - retrieval - scientific-papers - arxiv - research-library datasets: - PeytonT/1m_papers_text --- # Generated from Research Library: https://github.com/peytontolbert/Research_Library # 1M Paper Embedding Model 1M is a LoRA adapter for `allenai/scibert_scivocab_uncased` trained for scientific paper retrieval inside the Research Library project. It embeds paper queries and metadata cards so a user can search, rank, and navigate papers by title, abstract, category, and author metadata. This repository contains the PEFT adapter only. Load it on top of `allenai/scibert_scivocab_uncased`. ## Intended Use - Encode paper search queries for retrieval. - Encode paper metadata records for nearest-neighbor search. - Rank candidate papers in a research-library interface. This is not a generative model and should not be used to synthesize paper text. ## Training Data The adapter was trained from `PeytonT/1m_papers_text`, a 1M-paper full-text and metadata dataset. Training used the metadata fields available in the local Research Library pipeline: - `title` - `abstract` - `categories` - `authors` The training stream covered one clean full epoch over the 1M-paper corpus using positive and negative contrastive pairs. ## Training Procedure - Base model: `allenai/scibert_scivocab_uncased` - Adapter: LoRA - Task type: `FEATURE_EXTRACTION` - LoRA rank: `8` - LoRA alpha: `32` - LoRA dropout: `0.05` - Target modules: `query`, `value` - Objective: contrastive metadata retrieval - Batch size: `512` - Max source tokens: `256` - Precision: `bf16` - Optimizer: `adamw` - Learning rate: `1e-4` - Warmup steps: `1000` - Training steps: `3907` - Epochs: `1.0` - Final train loss: `0.0250` - Hardware: single H100-class GPU ## Usage ```python import torch import torch.nn.functional as F from transformers import AutoModel, AutoTokenizer from peft import PeftModel repo_id = "PeytonT/1m-paper-embedding-model" base_id = "allenai/scibert_scivocab_uncased" tokenizer = AutoTokenizer.from_pretrained(repo_id) base = AutoModel.from_pretrained(base_id) model = PeftModel.from_pretrained(base, repo_id) model.eval() def embed(texts): batch = tokenizer( texts, padding=True, truncation=True, max_length=256, return_tensors="pt", ) with torch.no_grad(): outputs = model(**batch) mask = batch["attention_mask"].unsqueeze(-1) pooled = (outputs.last_hidden_state * mask).sum(dim=1) / mask.sum(dim=1).clamp_min(1) return F.normalize(pooled, dim=1) query = embed(["retrieval augmented generation for scientific literature"]) docs = embed([ "Title: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\nCategories: cs.CL", "Title: Quantum error correction with superconducting qubits\nCategories: quant-ph", ]) scores = query @ docs.T print(scores) ``` ## Limitations - The adapter is optimized for metadata retrieval, not full-text semantic chunk retrieval. - It depends on the SciBERT base model and PEFT adapter loading. - Training was contrastive and library-oriented; external benchmarks have not yet been run. - Metadata quality, missing abstracts, and noisy category labels can affect retrieval quality. ## Project Context This model is part of the Research Library system for exploring repositories and scientific papers through search, metadata views, paper graphs, and 3D universe visualizations. ## Framework Versions - PEFT `0.19.1`