🇹🇭 Thai 0.1B SLM (Base / Mid-Trained Model)
A modern, highly efficient 108 Million parameter (0.1B) Thai-English Small Language Model (SLM) trained on over 504 Million tokens across 6,500 training steps.
Developed by Jommarn, this model incorporates cutting-edge architectures inspired by Gemma 2 and DeepSeek-V3 to maximize reasoning and linguistic capability under a strict compact footprint.
🧠 Architectural Highlights
| Feature | Specification | Rationale |
|---|---|---|
| Parameters | 108,160,512 (~108M) | Extremely lightweight, runs on mobile & edge devices |
| Layers | 14 Transformer Blocks | Deeper network for strong semantic abstraction |
| Attention | Interleaved Local-Global | 7 Layers Sliding Window (512) + 7 Layers Global Attention (Gemma 2 style, saves 50% KV cache) |
| GQA | 6 Query Heads, 2 KV Heads | Grouped Query Attention for fast, memory-efficient decoding |
| MTP | Multi-Token Prediction | Dual-loss prediction ($T+1$ and $T+2$) inspired by DeepSeek-V3 |
| Activations | SwiGLU & RMSNorm | Superior gradient propagation and numerical stability |
| Embeddings | Tied Input-Output (16K) | Weight sharing saves 12.5M parameters |
| Context Length | 1,024 tokens (Window 512) | Optimized for compact dialogues and reasoning |
📈 Training Progression
- Pre-training Phase 1 (Wiki Pre-train): 3,800 Steps (~185M tokens). Loss dropped from 9.7 down to 2.1.
- Mid-training Phase 2 (Step 1 to 4,500): Resumed with high-quality Thai web streams (
WangchanLION-Web), Thai food, and tech news. Total accumulated tokens reached >406M tokens. - Mid-training Phase 2 Expansion (Step 4,500 to 6,500): Expanded with gentle cosine decay to Step 6,500. Total accumulated tokens reached >504M tokens, with next-token loss
L(T+1)reaching 0.90 - 1.05!
💻 How to Run Inference
Use the standalone PyTorch modeling code modeling_0_1b.py with Hugging Face weights:
import torch
from transformers import PreTrainedTokenizerFast
from huggingface_hub import hf_hub_download
import sys
# 1. Load Tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("Jommarn/thai-0.1b-tokenizer")
# 2. Download Model Checkpoint
ckpt_path = hf_hub_download(repo_id="Jommarn/thai-0.1b-slm", filename="midtrain_step_6500/model_checkpoint.pt")
checkpoint = torch.load(ckpt_path, map_location="cpu")
# 3. Initialize Model Architecture
from modeling_0_1b import Thai0_1BForCausalLM
model = Thai0_1BForCausalLM(checkpoint["config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
# 4. Generate
prompt = "กรุงเทพมหานครเป็นเมืองหลวงและศูนย์กลางทางเศรษฐกิจของประเทศไทย โดยมี"
input_ids = tokenizer.encode(prompt, return_tensors="pt")
with torch.no_grad():
output = model.generate(input_ids, max_new_tokens=150, temperature=0.7, top_k=30)
print(tokenizer.decode(output[0].tolist()))
📜 License
Apache-2.0. Free for open research and commercial use.