--- license: apache-2.0 --- # BanBTP Model Family | Model | Base Architecture | Parameters | Context | Status | Notes | |-------|-------------------|------------|---------|--------|-------| | BanBTP 124M (V1) | GPT-2 (from scratch) | 124M | 1024 | Conversational | Good for some conversation, but not everything. | | BanBTP 783M (V2) | DialoGPT | 783M | 1024 | Repetitive | WORSE than v1 (124M) | | BanBTP V3 | GPT2 (from scratch) | 19M | 128 | Conversational AND repetitive :3 | Small model. | | BanBTP V4 | GPT2 (from scratch) | 41M | 128 | Mostly garbled outputs. | Larger but worse in quality | | BanBTP V5 | GPT2 (from scratch) | 54M | 256 | Aims to be better | Aims to outperform V1,2,3 and 4! It is a combination of V1,2,3 and 4 into one! It is distilled from all 4 + some gemma 3 models (270M) and gpt2 models. (GPT2-large) and the BanBTP distilled dataset too. | | BanBTP V6 | GPT2 (from scratch) | 54M | 256 | Aims to be better than V5 | Distilled from 124M and V5 as well as tinyllama 1.1B and GPT2-large. | Will try to outperform 124M. | | BanBTP V7 | GPT2 (from scratch) | 67M | 256 | Experiemental RAG release. proof of concept.| Not gonna outperform V6 in terms on knowledge but maybe in terms of RAG. | | BanBTP V8 | GPT2 (from scratch) | 110M | 256 | Aims to be a good chatbot. | Chatbot | All BanBTP models are trained on the BanBTP dataset. this is internet data. V5 however is trained on the BanBTP distilled dataset (from various models), so model will be slightly better than V4,3,2 and 1 but V6 will be slightly better than V5. So a edited response could be: "All models from V1-4 are based on a closed source BanBTP dataset" while V5 is based on distilled dataset from all these models." V6 will be based on V5 distilled, but will not use the BanBTP dataset. instead, we will use V5 and finetune it on BanBTP 124M as that one was the highest quality as well as GPT2-large and tinyllama 1.1b and a updated higher quality BanBTP dataset. Please note: V3-V4 are trained on same BanBTP dataset, however 124M was finetuned a lot better than the V3-4. V3-4 are more of test models than anything, to experiment to see if BanBTP can be well on smaller sizes. Turns out not. V5 is a mini BanBTP 124M. V6 will improve upon V5's training data and be more like BanBTP 124M in the sense it aims to beat it while being a tad bit smaller! V7 completely rewrites BanBTP. It is trained on BanRAG dataset. This model focuses BanBTP on RAG forcing the model to think logically in RAG making the RAG the data. It is a experimental model more than anything. Though, BanBTP models are pretty experimental anyways. :P V8 will aim to improve upon V7 and V6's core arch. by merging them together. V7 distilled into V8 and model given 76M parameters. V9 is the last BanBTP model. (It will have tool calling, RAG and conversational.)> {EDIT: it will not have tool calling or RAG. it will just be conversational.} EDIT for V7: As it turns out, V7 is unable to do RAG properly, so V8 will be completely rebuilt not on RAG but just on being a chatbot.
