Bangla Gamba

BanglaGamba

Huggingface compatible version of Bangla-Gamba Base model.

Model Details

Property	Value
Parameters	~200M
Context Length	2048
Architecture	Hybrid Mamba-3 + GQA (1:1 interleaved)
Primary Language	Bengali
Secondary Language	English
Training Tokens	~9.62B
Training Dataset	ahmed-farhanur-rashid/bn-foundational-pretrain-corpus
License	CC BY-NC-SA 4.0

Resources

• GitHub Repository: ahmed-farhanur-rashid/bangla-gamba
• Training Dataset: datasets/ahmed-farhanur-rashid/bn-foundational-pretrain-corpus

Requirements

This model uses a custom architecture and tokenizer implementation. Loading requires enabling trust_remote_code=True.

Install the required dependencies:

pip install transformers torch mamba-ssm causal-conv1d bnunicodenormalizer

Note: The model was trained on text normalized using bnunicodenormalizer in data pipeline. Running without it may significantly degrade generation quality.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "ahmed-farhanur-rashid/bangla-gamba"
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()
prompt = "বাংলাদেশের রাজধানী"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Architecture

Unlike conventional Transformer-only language models, BanglaGamba combines state-space modeling with attention mechanisms:

• Mamba-3 (State Space Model): Efficient linear-time sequential modeling and long-range state recurrence without quadratic attention overhead.
• Grouped Query Attention (GQA): Improves inference efficiency through optimized key-value caching, enhanced with per-head QK-Norm and RoPE.
• SwiGLU FFN: Interleaved in every block for non-linear representation capacity.

This hybrid design aims to balance computational efficiency with strong language modeling performance.

Related Models

BanglaGamba is part of a family of Bengali foundation language models.

Model	Architecture	Description
BanglaGamba	Mamba-3 + GQA	Hybrid state-space and GQA architecture optimized for efficient Bengali language modeling.
BanglaGSG	GDN + SWA + GQA	Sibling hybrid architecture trained on the same corpus.

• BanglaGSG: tasmin-jahan/bangla-gsg

Limitations

• The model expects text normalized using bnunicodenormalizer, consistent with the preprocessing pipeline used during training.
• Loading requires execution of custom Python modules (modeling_banglagamba.py, configuration_banglagamba.py, tokenization_banglagamba.py) via trust_remote_code=True.
• While primarily trained for Bengali, English support is intended mainly for multilingual understanding and translation-related capabilities.
• As with other large language models, outputs may occasionally be inaccurate or reflect biases present in the training data.

Citation

If you use BanglaGamba in your research, please cite the model:

@misc{banglagamba2026,
  title        = {BanglaGamba},
  author       = {Ahmed Farhanur Rashid},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ahmed-farhanur-rashid/bangla-gamba}}
}