Add TinyBrainBot-100M-v3-Base and TinyBrainBot-25M-v4-Base (TinyBrainBot)
#214
by nkthebass - opened
Adds two base models from TinyBrainBot (new org entry), both pretrained from scratch, open weights, final training checkpoints (not merges). Both load as a standard Transformers LlamaForCausalLM with tied embeddings and no custom code.
TinyBrainBot-100M-v3-Base - 100,092,672 parameters (75.5M non-embedding + 32k-vocab tied embedding), ~25.7B tokens.
| Benchmark | Score |
|---|---|
| ARC-Easy (acc_norm) | 54.67 |
| ARC-Challenge (acc_norm) | 30.12 |
| HellaSwag (acc_norm) | 32.62 |
| PIQA (acc_norm) | 66.21 |
| ArithMark-3 (length-normalized) | 37.40 |
TinyBrainBot-25M-v4-Base - 30,554,496 parameters (25.2M non-embedding + 14k-vocab tied embedding), ~11B tokens.
| Benchmark | Score |
|---|---|
| ARC-Easy (acc_norm) | 42.09 |
| ARC-Challenge (acc_norm) | 24.91 |
| HellaSwag (acc_norm) | 28.86 |
| PIQA (acc_norm) | 60.23 |
| ArithMark-3 (length-normalized) | 35.20 |
Zero-shot, full test sets, lm-eval 0.4.13 (float32); ArithMark-3 with the official bencharithmark-3.py (float32).
To reproduce:
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-100m-v3-base,dtype=float32 --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-100m-v3-base --dtype float32
(same with nkthebass/tinybrainbot-25m-v4-base)
Great stuff! Impressive models and welcome to the leaderboard!
Datdanboi25 changed pull request status to merged