Upbringing

An interactive tour of LLM post-training

Upbringing

A pretrained language model doesn't answer questions; it continues text. Post-training turns it into an assistant, and then into something that can reason. This page steps through real post-training runs on open models, checkpoint by checkpoint. Drag the slider to watch one learn to answer.

YouWho are you?
Base model, before post-training

drag or tap a checkpoint

Post-training typically runs in stages. Each section below covers one, with its own model, training run and evaluation.

1 From base model to assistant SFT then DPO on OLMo-2 1B, following AI2's recipe. Scrub through every checkpoint. SFT + DPO 2 Learning from preferences Judge answer pairs and see the reward model your choices imply, then watch DPO amplify a labeler's taste. RLHF-style preference data 3 Reinforcement learning with a verifier GRPO on Countdown puzzles: Qwen2.5-3B learns from a program that checks its answers. RLVR

By Dennis Koshta · October 2026 · Code on GitHub

1

From base model to assistant

We post-trained OLMo-2 1B, a fully open base model, with AI2's published recipe: supervised fine-tuning (SFT) on 866k example conversations, then direct preference optimization (DPO) on pairs of answers labeled better and worse. We saved dozens of checkpoints along the way, densest at the start, where the model changes fastest.

Pick a prompt and scrub through training. A dot after a step number marks a checkpoint with a note.

Base model
You
Model
Each token is shaded by how much more likely it is at this checkpoint than under the base model. Hover or tap a token for the probabilities.

Benchmarks

At the selected checkpoint, against the base model and AI2's released DPO model (the same recipe at full budget). The trend lines are clickable.

Diversity: “pick a random number”

Diversity: “tell me a joke”

Samples are drawn from 300 real generations (temperature 1) recorded at the selected checkpoint.

Under the hood

2

Learning from preferences

Preference tuning trains on pairs of answers where a judge marked one as better. It's the core of RLHF (reinforcement learning from human feedback), the method behind ChatGPT-style assistants. Whatever the judges consistently reward, the model learns, including preferences they never meant to express. Try judging a few pairs.

First, what makes an answer good to you? Pick up to two.

Training on a labeler's judgments

Three simulated labelers each follow one hidden rule. Each round, a labeler judges 16 fresh pairs of the model's own answers, and the model takes a LoRA DPO update on those judgments. Pick a labeler and train it round by round.

not trained yet
You
Model

Drift in held-out answers

Averages over answers to 24 prompts the labeler never judged; the labeler's target trait is highlighted.

Recovered reward model

A Bradley–Terry fit to the labeler's judgments so far, the same fit used for your choices above.

Under the hood
  • Each round: 16 new questions, two sampled answers each, the judge's pick, then a LoRA DPO update (β = 0.1, 3 epochs). The reference model is the model at the start of the round (iterative DPO), so changes compound.
  • Classic RLHF first trains a separate reward model on the preference pairs, then runs reinforcement learning against it. DPO learns from the same kind of pairs directly, without the separate reward model. We use DPO throughout.
  • “What the picks reveal” is a Bradley–Terry fit: a logistic regression on the differences between the two answers' traits. Traits are counted by simple rules (length in tokens, list items and headers, hedge words, confident words, exclamation marks).
  • The replays start from our own SFT checkpoint (step 6,500 from section 1). They were generated on a laptop GPU (RTX 4060, about 100 seconds per round) by this script. Live training from your own judgments is not built.

3

Reinforcement learning with a verifier

Here the training signal comes from a program instead of people. Qwen2.5-3B, a base model, is given Countdown puzzles (combine the numbers to reach a target), and a verifier scores every attempt: 1 for a correct equation, 0.1 for one that is merely well-formed. It is never shown a solution. This is RLVR, reinforcement learning from verifiable rewards, the approach behind reasoning models like DeepSeek-R1, trained here with GRPO.

Try one yourself

Use each number exactly once, with + − × ÷ and parentheses. Your answer is scored by the same verifier as the model's.

Training curves

Measured on puzzles it never practiced on. Hover any chart to read a value.

Follow one puzzle through training

Four samples for the same held-out puzzle at each evaluation checkpoint. The prompt asks for reasoning inside <think> tags and the result inside <answer> tags; underlines mark reasoning phrases.

base model
Under the hood
  • Algorithm: GRPO. Each round samples 8 attempts for each of 32 puzzles and reinforces attempts that score above their group's average. Reward: 1 for a correct equation, 0.1 for a well-formed answer. Learning rate 1e-6, no KL penalty, 900 optimizer steps, full-parameter training on one H200.
  • Puzzles come from the Countdown dataset used by TinyZero. The last 256 are held out and never trained on; all 256 are solvable (checked by brute force). Accuracy during training uses those 256 puzzles × 4 tries at temperature 1 (1,024 answers per point).
  • “How it talks” counts phrases with simple patterns. It shows how the model phrases its reasoning, not whether each check is valid.

Methods

How this was made

Everything here is reproducible from the repo: one command per stage, pinned configurations, and tests on the parts that decide correctness. Every place we cut corners to fit the compute budget is listed below.