--- language: - multilingual base_model: - google/t5gemma-2-4b-4b pipeline_tag: text-ranking datasets: - KaLM-Embedding/KaLM-embedding-finetuning-data - Shitao/bge-m3-data tags: - reranker - encoder-decoder - FBNL - Retrieval - RAG license: apache-2.0 library_name: transformers --- KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking We present `KaLM-Reranker-V1`, a fast but not late-interaction (FBNL) reranker that decouples query and passage computation while retaining expressive relevance modeling. Built on an encoder-decoder architecture, KaLM-Reranker-V1 uses the encoder to pre-encode passages with Matryoshka embedding pooling, while the decoder models the system instruction, user instruction, and query intent; cross-attention then captures relevance between the query context and passage representations. This design makes KaLM-Reranker-V1 efficient through decoupled passage encoding, yet not late interaction, by preserving rich relevance modeling through cross-attention. We instantiate KaLM-Reranker-V1 in three sizes, `Nano`, `Small`, and `Large`, with `0.27B`, `1B`, and `4B` activated parameters, respectively.  Extensive experiments on BEIR, MIRACL, and LMEB show that the KaLM-Reranker-V1 series achieves competitive reranking performance compared with strong industrial rerankers while significantly reducing online overhead. # Model Details | Models | Activated Params. | Non-Embedding Params. | Embedding Params. | #Layers | Sequence Length | Document Token Dim. | MEP Support | Instruction Aware | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | [KaLM-Reranker-V1-Nano](https://huggingface.co/KaLM-Embedding/KaLM-Reranker-V1-Nano) | 0.27B | 100M | 168M | 18 | 128K | 640 | 1x-32x | Yes | | [KaLM-Reranker-V1-Small](https://huggingface.co/KaLM-Embedding/KaLM-Reranker-V1-Small) | 1B | 698M | 302M | 26 | 128K | 1152 | 1x-32x | Yes | | [KaLM-Reranker-V1-Large](https://huggingface.co/KaLM-Embedding/KaLM-Reranker-V1-Large) | 4B | 3209M | 675M | 34 | 128K | 2560 | 1x-32x | Yes | # Prompt Template ```python f" : {document}" ``` ```python ( f" user\n" f"Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".\n\n" f" : {task_instruction}\n" f" : {query} \n" f" model\n\n\n\n" ) ```  # Evaluation ## BEIR On BEIR, KaLM-Reranker-V1 achieves state-of-the-art performance, on par with strong industrial models such as the Qwen3-Reranker series.  ## MIRACL On MIRACL, despite not being extensively trained on multilingual data, KaLM-Reranker-V1 still shows excellent reranking performance.  ## LMEB On LMEB, reranking models demonstrate a clear advantage, with even the 0.27B Nano model remaining competitive with 7–12B embedding models.   # Usage ## Using transformers ```python import argparse from typing import Optional def optional_positive_int(value: str) -> Optional[int]: if value.lower() == "none": return None try: parsed = int(value) except ValueError as error: raise argparse.ArgumentTypeError( "must be a positive integer or 'none'" ) from error if parsed argparse.ArgumentParser: parser = argparse.ArgumentParser( formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) parser.add_argument( "--model", default="KaLM-Embedding/KaLM-Reranker-V1-Large", help="Hugging Face model ID or local checkpoint path.", ) parser.add_argument( "--device", default=None, help="Inference device, such as 'cuda', 'cuda:0', or 'cpu'.", ) parser.add_argument( "--dtype", default=None, choices=("bfloat16", "bf16", "float16", "fp16", "float32", "fp32"), help="Model parameter dtype. By default, use BF16 on CUDA and FP32 on CPU.", ) parser.add_argument( "--batch-size", type=int, default=32, help="Number of query-document pairs scored per inference batch.", ) parser.add_argument( "--query-max-length", type=int, default=512, help=( "Maximum tokens in the raw query before it is inserted into the " "decoder prompt; prompt tokens are not included in this limit." ), ) parser.add_argument( "--reranker-max-length", type=int, default=1024, help=( "Maximum encoder tokens for ' : {passage}'. This is not a " "combined query-document context limit." ), ) parser.add_argument( "--chunk-size", type=optional_positive_int, default=4, metavar="N|none", help=( "Number of encoder token hidden states per mean-pooled chunk; use " "'none' to disable encoder chunk pooling." ), ) return parser def main() -> None: args = build_parser().parse_args() from kalm_reranker import KaLMReranker reranker = KaLMReranker( args.model, device=args.device, dtype=args.dtype, batch_size=args.batch_size, query_max_length=args.query_max_length, max_length=args.reranker_max_length, chunk_size=args.chunk_size, ) query = "What is the capital of China?" documents = [ "The capital of China is Beijing.", "Gravity attracts bodies toward one another.", ] instruction = "Given a query, retrieve documents that answer the query." pairs = [(query, document) for document in documents] print("scores:", reranker.predict(pairs, instruction=instruction)) print("rankings:", reranker.rank(query, documents, instruction=instruction)) if __name__ == "__main__": main() ''' scores: [0.9998205304145813, 4.7850949158601e-06] rankings: [{'corpus_id': 0, 'score': 0.9998205304145813}, {'corpus_id': 1, 'score': 4.7850949158601e-06}] ''' ``` ## Using vLLM An experimental single-GPU adapter is available for offline `LLM.classify()` reranking and optional FastAPI serving. It reuses the original checkpoint without adding or modifying model weights. The adapter has been validated with Python 3.12, vLLM 0.19.1, Transformers 5.6.2 and CUDA BF16: ```bash conda create -n kalm-vllm python=3.12 -y conda activate kalm-vllm pip install "vllm==0.19.1" "transformers==5.6.2" hf download KaLM-Embedding/KaLM-Reranker-V1-Large \ --local-dir ./KaLM-Reranker-V1-Large pip install ./KaLM-Reranker-V1-Large/vllm_support --no-deps export VLLM_PLUGINS=kalm_t5gemma2 ``` Offline Python: ```python from kalm_t5gemma2_vllm_plugin import KaLMVLLMReranker query = "What is the capital of China?" documents = [ "The capital of China is Beijing.", "Gravity attracts bodies toward one another.", ] with KaLMVLLMReranker( "KaLM-Embedding/KaLM-Reranker-V1-Large", query_max_length=512, document_max_length=1024, encoder_chunk_size=4, ) as reranker: print(reranker.rank(query, documents)) ``` Offline CLI: ```bash kalm-vllm-rerank --return-margin ``` To deploy the online service, install the HTTP dependencies and keep the server running in the first terminal: ```bash pip install "fastapi>=0.136, =0.46,
