Teaching Phi-4-mini the dotnet/runtime codebase

Milos Kotlar

kotlarmilos

Follow

The short version

I fine-tuned Phi-4-mini-instruct to generate context-aware code review suggestions for pull requests in dotnet/runtime, a large open-source codebase. I built a custom dataset from the repository's own pull requests, trained the model with LoRA under 4-bit quantization on a single A100, and served the result as a hosted demo. About 10,000 samples and roughly four hours of training were enough to shift the model toward the project's own review style. This is the applied end of my series. It sits next to the from-scratch gpt2-nano and the reasoning post-training, and it is the one that ships.

One codebase has its own dialect

A general small model knows a little about everything and the specifics of nothing. A big codebase has its own dialect. Component names, triage labels, recurring failure modes, and reviewer conventions that a general model has never seen. I wanted to find out whether supervised fine-tuning on a repository's own pull requests moves a small model toward that dialect, and I wanted the answer as artifacts I can rerun rather than a claim I have to trust.

Turning pull requests into a timeline

I built the dataset by parsing pull requests into structured sequences. Each pull request becomes a timeline. The prompt holds the PR title, the body, the labels, and every event up to a given commit, and the completion is the code diff at that point. That framing turns the messy history of a PR into a clean prompt-and-completion pair, and it is where most of the outcome is decided. I published the dataset so the whole thing can be regenerated.

Training used LoRA with 4-bit quantization on a single A100. The roughly 10,000 samples took about four hours. I chose Phi-4-mini because its small size keeps the loop cheap enough to iterate on. I then deployed an inference endpoint as a hosted space, so the result can be queried directly instead of only described.

What the tuned model writes now

The fine-tune produced code review completions that are more relevant and better structured than the base model on the same prompts. Comparing the base and the tuned model side by side shows a clear improvement in how closely the output matches the repository's review style, even though this prototype did not use formal instruction formatting during training. The deliverable is the end-to-end artifact set. A domain dataset, a fine-tuned checkpoint, and a live space.

What the run taught me

Building this end to end taught me a set of concrete things that only show up once you run the full loop.

• About 10,000 samples, or even fewer, is enough to fine-tune a model in a way you can feel. I expected to need far more.
• A single A100 fine-tuned the 4-bit quantized model in about four hours, which keeps the whole experiment in the range of a single afternoon.
• Always use the model's own predefined chat format, the <|system|>...<|tool|>... template. Training outside that format leaves quality on the table.
• Fine-tune the LoRA weights under quantization, then plug them into a full-precision model at inference. You get the cheap training without paying the quantization cost when it matters.
• Inference on a T4 takes about 20 seconds for 256 tokens, which sets the expectation for what a live demo feels like.
• Filter the dataset down to high-quality short prompts and completions. Otherwise the completion gets truncated to the max token limit during training, and the model learns from cut-off examples.

The takeaway is simple. Small-scale fine-tuning on around 10,000 prompt-and-completion pairs can meaningfully change how a model behaves, and the data shaping is the part that decides how far it moves.

Where I would take it next

The obvious next step is better context at inference rather than more training. I would filter the dataset and repeat the fine-tune, then train a separate encoder to find relevant functions in the source, index every function so I can retrieve similar ones, and hand a foundational model the PR title and body, the relevant functions from the encoder, the similar functions from retrieval, and the fine-tuned suggestions together. The fine-tune is the start of the system, not the whole of it.

References

1. Microsoft. Phi-4 Technical Report. arXiv:2412.08905, 2024.
2. Ouyang, L. et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
3. Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.

Code, model, dataset, and demo.

• Code: phi-4-mini-dotnet-runtime
• Model: phi-4-mini-dotnet-runtime
• Dataset: dotnet-runtime
• Demo: space