aaro765/BanBTP-comparison

BanBTP Model Family

Model	Base Architecture	Parameters	Context	Status	Notes
BanBTP 124M (V1)	GPT-2 (from scratch)	124M	1024	Conversational	Good for some conversation, but not everything.
BanBTP 783M (V2)	DialoGPT	783M	1024	Repetitive	WORSE than v1 (124M)
BanBTP V3	GPT2 (from scratch)	19M	128	Conversational AND repetitive :3	Small model.
BanBTP V4	GPT2 (from scratch)	41M	128	Mostly garbled outputs.	Larger but worse in quality
BanBTP V5	GPT2 (from scratch)	54M	256	Aims to be better	Aims to outperform V1,2,3 and 4! It is a combination of V1,2,3 and 4 into one! It is distilled from all 4 + some gemma 3 models (270M) and gpt2 models. (GPT2-large) and the BanBTP distilled dataset too.
BanBTP V6	GPT2 (from scratch)	54M	256	Aims to be better than V5	Distilled from 124M and V5 as well as tinyllama 1.1B and GPT2-large.
BanBTP V7	GPT2 (from scratch)	67M	256	Experiemental RAG release. proof of concept.	Not gonna outperform V6 in terms on knowledge but maybe in terms of RAG.
BanBTP V8	GPT2 (from scratch)	110M	256	Aims to be a good chatbot.	Chatbot

All BanBTP models are trained on the BanBTP dataset. this is internet data. V5 however is trained on the BanBTP distilled dataset (from various models), so model will be slightly better than V4,3,2 and 1 but V6 will be slightly better than V5. So a edited response could be: "All models from V1-4 are based on a closed source BanBTP dataset" while V5 is based on distilled dataset from all these models." V6 will be based on V5 distilled, but will not use the BanBTP dataset. instead, we will use V5 and finetune it on BanBTP 124M as that one was the highest quality as well as GPT2-large and tinyllama 1.1b and a updated higher quality BanBTP dataset.

Please note: V3-V4 are trained on same BanBTP dataset, however 124M was finetuned a lot better than the V3-4. V3-4 are more of test models than anything, to experiment to see if BanBTP can be well on smaller sizes. Turns out not.

V5 is a mini BanBTP 124M. V6 will improve upon V5's training data and be more like BanBTP 124M in the sense it aims to beat it while being a tad bit smaller!

V7 completely rewrites BanBTP. It is trained on BanRAG dataset. This model focuses BanBTP on RAG forcing the model to think logically in RAG making the RAG the data. It is a experimental model more than anything.

Though, BanBTP models are pretty experimental anyways. :P

V8 will aim to improve upon V7 and V6's core arch. by merging them together. V7 distilled into V8 and model given 76M parameters.

V9 is the last BanBTP model. (It will have tool calling, RAG and conversational.)> {EDIT: it will not have tool calling or RAG. it will just be conversational.}

EDIT for V7: As it turns out, V7 is unable to do RAG properly, so V8 will be completely rebuilt not on RAG but just on being a chatbot.