Research and data
Collection
Controlled experiments and datasets. • 5 items • Updated
Do data-mix rankings transfer across scale?
A three-arm controlled study at 100M parameters. Models identical in every respect except their training data, measured on one shared held-out set, seven neutral domain slices and fifteen standard benchmarks.
Status: complete. The methodology, configs and validation-set construction were published before any arm finished. Arm C was iterated twice; both iterations are reported here with the reasoning for the second stated rather than hidden, and iteration 2 is the one shipped as weights. A data-ordering bug that cost a full 25-hour run is documented with what it measured.
These are not usable models. 138M parameters on ~5B tokens is heavily undertrained by design. They exist to answer a methodological question, and are published so the answer can be checked. No GGUFs are provided: nothing here is meant for deployment.
The question
Small-scale proxy runs are the standard way to choose a data mixture, and whether small-scale rankings survive scaling is actively studied, not an open question: DoReMi tunes domain weights on a 280M proxy and applies them at 8B, RegMix fits a regression from many small runs, Data Mixing Laws and Aioli model mixture performance directly, and DataDecide measures how well small-scale benchmark rankings predict larger-scale ones across thousands of runs.
This pilot does not add to that literature. It is a local replication with a narrower purpose: before committing a 430M-parameter budget, establish whether these three candidate corpora are distinguishable at 100M, with what measurement machinery, and at what cost. The transfer question is what a 430M run would test; nothing here answers it.
These pilots isolate one variable: data. All arms share a single stable-then-decay schedule. A 430M run would use a more complex multi-stage schedule, so nothing here is a claim about that schedule.
The arms
Identical architecture, hyperparameters, seed, schedule shape and precision. Only the data differs.
| Folder | Arm | Data | Role |
|---|---|---|---|
armA/ |
A | the 350M V3 stable pretrain mix, condensed to ~5B tokens | the incumbent approach |
armB/ |
B | IMU-1's released corpora, re-tokenised to the shared vocab | the control |
armC/ |
C (iteration 2) | redesigned mix, web restored to 44%, dense-fact and QA raised | the treatment |
Arm C went through two iterations, and both are reported throughout. Iteration 1 ran the same
redesign at 28% web; iteration 2 raised web to 44% and added dense-fact and QA data. Only
iteration 2 is shipped as weights, because it is the better model and the one the arm-C claims
rest on - but iteration 1 appears in every results table below and its raw eval output is in
results/pilot_armC2.json, since the web-saturation finding is that comparison. Reporting the
iteration count is the point: the mix was refined until it produced a good result, which is
exactly the practice under test.
Arm A's mix is the stable-phase pretrain mix, not the anneal/extend mix used late in that
model's training. At 5B tokens the extend mix would place reasoning-distill at 5.2 epochs and
turn the run heavily synthetic.
Arm C is post-hoc by construction. Iteration 1 was designed after diagnosing where A and B fail; iteration 2 after seeing iteration 1 lose the primary metric, targeting the one variable that diagnosis pointed at: web share. What keeps this honest is that any arm-C advantage must survive a scale change, which this study does not attempt and cannot claim.
Arm C, iteration 2 — the published mix
| Source | Share | Source | Share | |
|---|---|---|---|---|
| fineweb-edu-10bt | 33.0% | qa-plain | 3.0% | |
| wiki-dense | 12.0% | gutenberg-clean2 | 2.0% | |
| dclm-clean | 11.0% | wikipedia-english | 1.5% | |
| the-stack-smol | 7.5% | code-pyjs | 1.5% | |
| finepdfs-edu | 7.0% | cosmopedia-openstax | 1.5% | |
| megamath-web-pro | 6.0% | qa-distill | 1.5% | |
| cosmopedia-stanford | 5.0% | facts-plain | 1.5% | |
| finemath-4plus | 3.0% | cosmopedia-auto-math | 1.0% | |
| capability-distill | 1.0% | |||
| reasoning-distill | 0.5% | |||
| bookcorpus-clean | 0.5% |
By group, against the other arms:
| Group | A | B | C iter1 | C iter2 |
|---|---|---|---|---|
| web | 57.6 | ~37 | 28.0 | 44.0 |
| reference | 8.0 | ~11 | 13.0 | 13.5 |
| code | 3.1 | ~9 | 9.0 | 9.0 |
| math | 0.2 | ~23 | 18.0 | 9.0 |
| synthetic-edu | 8.9 | — | 11.0 | 7.5 |
| qa-like | 4.5 | — | 6.0 | 7.0 |
| edu-PDFs | 0.0 | — | 9.0 | 7.0 |
| narrative | 17.6 | — | 5.0 | 2.5 |
| reasoning | 1.2 | — | 1.0 | 0.5 |
Arm A and C shares are measured token shares from their build plans. Arm B's column is
byte-share estimates from its download plan (pilot/armB_plan.json), grouped by hand — a
different basis, so treat it as approximate.
Iteration 2 added two sources no arm had ever used: facts-plain (short foundational facts
in varied phrasings) and capability-distill (cause-and-effect prose). Justification: the 100M
v3 base, an earlier model at this size that scored well against Supra2-100M, ran 8.74% QA-like
data against arm A's 4.46%. That is motivation, not evidence: whether the share caused those
scores was never tested. It also raised wiki-dense (Wikipedia article leads, the densest
factual text available here) from 8% to 12%, matching the share both annealed models in this
project used.
Shared configuration
hiddenSize 768 · nLayers 16 · nHeads 12 · nKvHeads 4 · headDim 64
ffnSize 2048 · vocabSize 49152 · tied embeddings · contextLength 1024
seqLen 1024 · microBatch 8 · gradAccum 8 · 2 GPUs = 131,072 tokens/step
lr 6e-4 · WSD schedule · warmup 800 · min_lr 0 · fp16 · grad-clip 1.0 · AdamW(0.9, 0.95) · wd 0.1
138.4M parameters (100.7M non-embedding + 37.7M tied embeddings)
| Run | Steps | Tokens | Anneal | Anneal share | LR decay fraction |
|---|---|---|---|---|---|
| A | 38,690 | 5.071B | 250M | 4.93% | 0.049 |
| B | 38,224 | 5.010B | 255M | 5.08% | 0.051 |
| C iter1 | 37,438 | 4.907B | 228M | 4.65% | 0.047 |
| C iter2 | 38,693 | 5.072B | 254M | 5.00% | 0.050 |
C iteration 2 was built to arm A's exact token budget, removing the 3.3% budget advantage A held over iteration 1. B and C-iter1 remain 1.2% and 3.3% below A; each arm trains on exactly the corpus its build produced rather than truncating to a common floor. The LR decay fraction is aligned to each arm's anneal fraction, so the decay phase and the anneal diet coincide.
Tokenizer: SmolLM2, 49,152 vocab, for every arm. IMU-1's corpora are released pre-tokenised at vocab 65,534 and were re-tokenised to the shared vocab, a 1.043x token-count ratio.
fp16, not bf16. On the V100s used here, bf16 stores weights with no fp32 master copy and RMSNorm gains never move off their initialisation. That bug cost an earlier 20B-token run in this project; the choice is deliberate.
Tied embeddings deviate from IMU-1, whose embeddings are untied. Identical across arms, so it cannot affect the comparison.
Metrics
Primary, fixed before any arm ran: held-out cross-entropy on a common 50M-token validation
set from OpenWebText2's test/val splits — already held-out text, not carved out of any
arm's training mix.
Secondary: seven neutral domain slices of 5M tokens each from the validation split of
monology/pile-uncopyrighted, which belongs to no arm's training data.
Tertiary: fifteen lm-eval tasks (EleutherAI lm-eval v0.4.13, 0-shot except GSM8K at 5-shot, n=2000 per task or the full test set where smaller). The task list was fixed before any arm was benchmarked and every task is reported for every arm, including the ones each arm loses.
Stated limitation: arm B is IMU-1's corpus, published pre-tokenised and therefore not auditable. Exact-document independence from the validation sets is verified for arms A and C and is unverifiable for arm B.
Results
Primary — held-out CE (OpenWebText2, 50M tokens; lower is better)
| Arm | CE | Perplexity |
|---|---|---|
| A — 350M V3 mix | 3.3855 | 29.53 |
| C iteration 2 | 3.3943 | 29.79 |
| B — IMU-1 data | 3.4169 | 30.47 |
| C iteration 1 | 3.4218 | 30.63 |
Secondary — neutral Pile slices (CE; lower is better)
| Slice | A | B | C iter1 | C iter2 |
|---|---|---|---|---|
| code | 2.0188 | 1.8908 | 1.7799 | 1.8003 |
| math | 1.9961 | 1.7917 | 1.7919 | 1.8117 |
| legal | 2.9803 | 2.9394 | 3.0079 | 2.9901 |
| medical | 3.0895 | 3.1849 | 3.1654 | 3.1611 |
| patents | 2.9623 | 2.9797 | 3.0078 | 2.9907 |
| reference | 2.7612 | 2.9784 | 2.8048 | 2.7765 |
| web | 3.3402 | 3.4456 | 3.4112 | 3.3860 |
| unweighted mean | 2.7355 | 2.7444 | 2.7098 | 2.7023 |
Tertiary — lm-eval, all fifteen tasks (±1 standard error)
acc_norm where the task provides it, acc otherwise, strict exact_match for GSM8K.
| Task | A | B | C iter1 | C iter2 |
|---|---|---|---|---|
| ARC-Easy | 49.2 ±1.1 | 48.4 ±1.1 | 45.1 ±1.1 | 49.2 ±1.1 |
| ARC-Challenge | 28.4 ±1.3 | 25.8 ±1.3 | 26.3 ±1.3 | 27.0 ±1.3 |
| SciQ | 73.3 ±1.4 | 67.0 ±1.5 | 67.4 ±1.5 | 70.5 ±1.4 |
| OpenBookQA | 29.0 ±2.0 | 30.4 ±2.1 | 28.8 ±2.0 | 30.8 ±2.1 |
| MMLU | 23.1 ±0.4 | 24.7 ±0.4 | 24.9 ±0.4 | 26.1 ±0.4 |
| HellaSwag | 37.0 ±1.1 | 37.1 ±1.1 | 38.5 ±1.1 | 36.5 ±1.1 |
| PIQA | 61.6 ±1.1 | 59.6 ±1.1 | 60.6 ±1.1 | 60.7 ±1.1 |
| WinoGrande | 51.8 ±1.4 | 49.8 ±1.4 | 52.9 ±1.4 | 52.6 ±1.4 |
| Social IQa | 38.3 ±1.1 | 36.2 ±1.1 | 36.0 ±1.1 | 37.5 ±1.1 |
| CommonsenseQA | 20.2 ±1.2 | 22.6 ±1.2 | 21.6 ±1.2 | 20.2 ±1.2 |
| LAMBADA (acc) | 33.7 ±1.1 | 25.4 ±1.0 | 28.6 ±1.0 | 27.9 ±1.0 |
| BoolQ | 61.1 ±1.1 | 56.8 ±1.1 | 61.0 ±1.1 | 56.1 ±1.1 |
| MathQA | 23.6 ±0.9 | 24.1 ±1.0 | 23.2 ±0.9 | 23.2 ±0.9 |
| GSM8K (5-shot) | 0.2 ±0.1 | 1.1 ±0.3 | 1.5 ±0.3 | 1.1 ±0.3 |
| arithmetic (10 subtasks, mean) | 1.08 | 1.12 | 1.03 | 1.40 |
LAMBADA perplexity: A 40.8, B 100.8, C iter1 73.6, C iter2 78.5.
What this shows
1. These three mixes are distinguishable at 100M on domain slices, but not in aggregate. Individual slices separate the arms by up to 0.24 nats, far above measurement noise. The aggregate numbers do not: A and C iteration 2 differ by 0.0088 on held-out CE, A and B by 0.0089 on the domain mean. With one seed per arm and no seed-variance estimate, gaps below roughly 0.03 nats are not resolvable here, so the aggregate ordering should be read as unranked.
2. No arm sweeps, and the ranking depends on the metric.
- A has the lowest primary held-out CE (3.3855 vs 3.3943), a gap too small to call at n=1, and real benchmark leads on LAMBADA (+5.8 over C iter2) and BoolQ (+5.0), both outside the error bars. SciQ (+2.8) and ARC-Challenge (+1.4) sit at or inside 2 SE.
- C iteration 2 has the lowest domain mean (2.7023) and the highest MMLU (26.1 ±0.4, 2.75 SE above the 25% chance line, marginal and one task), plus the arithmetic subtasks, and ties A on ARC-Easy and OpenBookQA. It wins no individual domain slice; it places second on almost all of them, which is how it takes the mean. The mean is unweighted over seven slices chosen by us, so reweighting or swapping slices can move that ranking.
- B wins math and legal, CommonsenseQA and MathQA.
The primary metric was named before the runs and is not renamed now that a different metric favours the treatment arm. A retains the primary metric; C iteration 2 is the best all-rounder.
3. Hypothesis, not a result: web share saturates and narrative carries the prose benchmarks. This comes from comparing C's two iterations, and that comparison is not a clean ablation — iteration 2 changed about eleven things at once: web 28→44, math 18→9, narrative 5→2.5, synthetic-edu 11→7.5, edu-PDFs 9→7, QA-like 6→7 with two new sources, reference 13→13.5, reasoning 1→0.5, tokens +3.3%, steps 37,438→38,693, anneal 4.65→5.00%. The deltas below belong to that whole change, not to web share alone:
| C iter1 (28% web) | C iter2 (44% web) | change | |
|---|---|---|---|
| held-out CE | 3.4218 | 3.3943 | −0.028 |
| Pile web slice | 3.4112 | 3.3860 | −0.025 |
| LAMBADA acc | 28.6 | 27.9 | −0.7 (no gain) |
| BoolQ | 61.0 | 56.1 | −4.9 |
| HellaSwag | 38.5 | 36.5 | −2.0 |
A 16-point web increase accompanied 0.028 nats of held-out gain and no LAMBADA movement. Across arms, what correlates with LAMBADA and BoolQ is narrative share — arm A runs 17.6% and leads LAMBADA 33.7 vs 27.9 and BoolQ 61.1 vs 56.1 — and both benchmarks reward long connected prose, which filtered web text does not supply. That is a plausible mechanism, not something this pilot tested: arm A differs from arm C in every group, and iteration 2's narrative cut was confounded with seven other changes. Likewise MMLU 23.1 → 26.1 and ARC-Easy +4.1 are associated with raising QA-like data to 7%, not shown to be caused by it.
Testing it would take single-variable runs — narrative alone, then web alone, at fixed budget. That is the obvious next experiment, and it is cheap at this scale.
4. Benchmarks are close to the floor at this scale. WinoGrande, CommonsenseQA and MMLU read at or near chance for all arms; GSM8K and arithmetic are at the floor. The loss metrics carry most of the signal, which is why they are primary.
Two data-pipeline bugs, and what they cost
Both were ordering bugs in the corpus builders. Neither changed what data an arm trained on; both changed the order. The second is a clean accidental measurement of what data order is worth.
Bug 1 — arm A, concatenated sources. The first build wrote each source contiguously, so the corpus ran 2.4B tokens of web text, then literary prose, then everything else. Caught from the loss curve (a 3.0 → 4.0 jump at a source boundary) before that run was used. Arm A was rebuilt with a global block shuffle and retrained.
Bug 2 — arm C, weighted round-robin per document. The rewritten builder interleaved sources
by drawing a document at a time, weighted by share. Long-document sources supply far more
tokens per draw, so they drained first: the corpus ran PDFs and wiki (4k–25k tokens/doc) → web
(1k) → short QA (300). Per-source totals were correct; the order was a curriculum. This was
not caught before training, and a full 25-hour run was spent on it.
What the bad order cost, same mix, same evals, order the only difference:
| Metric | drifted order | shuffled | difference |
|---|---|---|---|
| held-out CE | 3.5683 | 3.4218 | −0.147 |
| Pile mean CE | 2.8826 | 2.7098 | −0.173 |
| code slice | 2.3278 | 1.7799 | −0.548 |
The drifted run placed last on 6 of 7 slices — including code, where C had three times arm A's code share. That contradiction exposed the bug: a mix trade-off cannot make a source worse while tripling its share, but forgetting can.
The fix, now enforced. Corpora are built per source and then block-shuffled in 4,096-token blocks, bulk and anneal separately so the anneal stays last. Every training chain runs an order gate before step 0: document density (EOS per 1M tokens, ten windows per shard) must stay within 3x of the median across bulk shards, or the run aborts. Measured: shuffled A 1.57x, B 1.78x, C iter1 1.05x, C iter2 1.12x, drifted C 5.98x. Iteration 2's builder also selects sources by remaining token need rather than per document, fixing the cause rather than only the symptom.
A third fix: arm C iteration 1's first anneal contained zero qa-distill tokens against
a planned 6%, because the bulk had consumed that source and the anneal's document-skip ran past
its end. Small synthetic sources now get a second pass — one in pretraining, one in the anneal
(1.0–1.2 epochs total, inside the 3-epoch guard) — and both C anneals hit their planned
composition on every source.
Generations
samples.md holds greedy and sampled (temperature 0.8 and 0.9) continuations on 12 identical
prompts from all four runs, iteration 1 included even though its weights are not shipped: the part of this study a reader can judge without trusting any
number. At 138M on ~5B tokens the outputs are weak by construction — all arms loop, and all
confabulate. Two representative differences:
- "Water boils at a temperature of" — arm A answers 1000°C; B and both C iterations answer 100°C, and the C models continue into worked-exercise formatting, which is what a mix with finemath and QA data looks like from the outside.
- "She opened the door and found" — every arm degenerates into repetition, which is the narrative weakness the benchmark numbers above quantify.
Reproducing
Data specifications, build scripts with their epoch guards, the shuffle tool, the order gate and
the evaluation harness are in the project repository. The corpora are large; the build scripts
are deterministic given the same sources. Raw eval outputs for every arm are in results/.
lm-eval v0.4.13 · 0-shot (GSM8K 5-shot) · limit 2000 · batch 8 · fp32 · seeds 1234
Honest notes
- One seed per arm, and no seed-variance estimate. The single biggest limitation: without it, no gap smaller than roughly 0.03 nats can be called, which covers every aggregate comparison between A, B and C iteration 2. One seed supports "arm X achieved lower held-out loss than Y and Z under identical conditions". It does not support "mix X is better", and says nothing about 430M.
- The mechanism claims are hypotheses. Web saturation and the narrative effect come from a multi-variable change and cross-arm correlation, not from single-variable ablations.
- Arm C was iterated twice, both post-hoc, and its aggregate win is on the secondary metric, not the primary one.
- Narrative at 2.5% in iteration 2 is a known, deliberate weakness, recorded in the builder before training started, and the benchmarks confirmed it cost BoolQ and HellaSwag.
capability-distillcame in at 0.8% against a 1.0% plan — the source holds only ~34M tokens and ran dry; 0.2 points redistributed across the other sources.- Unequal token budgets for B (5.010B) and C iter1 (4.907B) against A's 5.071B. C iteration 2 matches A at 5.072B.
- Undertrained by design, ~36 tokens per parameter. These models are instruments, not artefacts.