Deep Learning Guide · 03
What AI Post-training Changes
A pretrained model has absorbed patterns from vast amounts of text, but it is not yet finished as a system that follows instructions, refuses dangerous requests, and answers in a requested format. Post-training uses examples, comparisons, rewards, and rules to steer an existing model toward more useful and controllable behavior.
The idea in three sentences
- Pretraining creates broad patterns and capabilities; post-training shapes when and how those capabilities are expressed.
- Demonstrations and preference comparisons translate human intent into data, but they also preserve the perspective of raters and whatever the measurement missed.
- Post-training can improve usefulness and safety, but it is not a final solution for truth, fairness, or controllability.
01 · What Post-training Means
The same base capability can be shaped into very different behavior.
Post-training is a family of adaptation procedures performed after pretraining. It aligns model output with intended uses such as conversation, summarization, coding, specialized domains, safety refusals, and tool calls.
- 01Pretraining
Learn a broad space of language and patterns
Next-token prediction learns relationships across language, code, and knowledge without defining one complete standard for a good answer.
- 02Post-training
Make desired answers and actions more likely
Demonstrations, preferences, rules, and verifiable rewards shape instruction following, format, safety, and specialist work.
- 03Deployment system
Add controls outside training
Prompts, retrieval, permission checks, filters, and human approval govern real use without changing model weights.
How are prompts different from post-training?
Prompt and context
Change the input for one request. They are fast and reversible, but instructions must be supplied repeatedly and behavior can be unstable.
Post-training
Changes default behavior across requests by adapting weights. It can be more consistent, but requires data, compute, and evaluation, and side effects can be harder to reverse.
02 · Specifying Behavior
An abstract request for ‘good AI’ must become observable material.
A model cannot read the developer’s intent directly. Desired and undesired behavior must be expressed through examples, comparisons, rules, and tests.
Demonstrations
Pair a prompt with an ideal response, directly showing content, tone, format, and refusal behavior.
Preference comparisons
Ask a person to select or rank the better of two or more responses to the same prompt.
Principles and policies
Write down safety, privacy, fairness, legal, and service boundaries for generation, critique, or evaluation.
Verifiable signals
Use outcomes such as passing code tests, correct mathematics, or format validation as rewards.
How one training item is made
- 01Select use case
- 02Write task or prompt
- 03Generate candidates
- 04Apply rating guide
- 05Choose, revise, and record
- 06Quality-check and split
03 · Supervised Fine-tuning
First, teach the model to imitate good demonstrations.
Supervised fine-tuning (SFT) places a human-written or reviewed target after a prompt and trains the model to assign higher probability to the target’s next tokens. Instruction tuning collects many tasks in natural-language instruction form to improve general instruction following.
Summarize this passage in three lines. A summary is…
1. Central claim 2. Main evidence 3. Remaining limitation
Answer directly in the requested format without continuing the prompt
Instruction format
Where the question ends and which part needs an answer
Response conventions
Conciseness, steps, tables, citations, and structured output
Role and domain
Vocabulary and patterns for support, document review, coding, or another target task
04 · Preferences & Reward
When one correct answer is hard to write, compare several candidates.
For dialogue, creative work, and safety—where there may be no single target—raters may find comparison easier than writing an ideal answer. Preference data can train the language model directly or train a reward model.
Explain a complex medical test result.
States uncertainty and need for care
Explains the general meaning without diagnosing the individual and suggests questions to confirm with a qualified clinician.
Confident but unsupported
Uses one value to declare a diagnosis and recommends treatment without sources or exceptions.
- 01
Generate candidates
The current model makes several answers to one prompt
- 02
Human comparison
A rater selects or ranks answers using a guide
- 03
Predict reward
A reward model estimates preference scores for new answers
- 04
Update policy
The language model is adjusted to produce high-reward answers more often
05 · Optimization Families
The same preference data can be applied through different methods.
Modern models combine methods, and implementation details vary across developers. These categories are a map of the central logic.
RLHF
Train a reward model from human comparisons, then use reinforcement learning to increase estimated reward. A constraint often keeps the policy from moving too far from the original model.
Is the system optimizing a loophole in the reward model?DPO
Increase the likelihood of chosen responses relative to rejected ones. This removes the separate reward-model and complex online reinforcement-learning stages.
Do comparison pairs represent real use?AI feedback
Using human-written principles, a model critiques and revises responses or generates preference labels. It scales data production but can also scale evaluator bias and error.
Do people meaningfully audit principles and outcomes?Verifiable rewards
Reward passing code tests, correct calculations, or valid format. Useful for clear tasks, but quality and safety outside the verifier may be missed.
Is passing the test the same as achieving the goal?06 · Efficient Adaptation
A model can be adapted without retraining every weight.
Full fine-tuning of a large model demands substantial memory and storage. Depending on purpose and resources, teams train only a small set of parameters or transfer behavior into a smaller model.
Full fine-tuning
Allows broad change but brings high compute and memory cost and a greater risk of damaging prior capabilities.
LoRA and adapters
Freeze base weights and train small matrices or modules. Several specialized adaptations can be stored relatively lightly.
Continued pretraining
Learn more text from medicine, law, or a particular language distribution before instruction and preference tuning.
Distillation
Train a smaller model from a larger model’s outputs or distributions. This lowers cost but may transfer capabilities and errors together.
07 · People, Rules & Power
Human labor and choices sit behind a model’s default manner.
Post-training data is not a naturally occurring answer key. The organization designing the work, the people writing guidance, and the workers producing and rating responses all shape behavior.
Who rated?
Are language, region, expertise, working conditions, and participant composition disclosed?
What was preferred?
When helpfulness, accuracy, harmlessness, and concision conflict, which criterion wins?
Who wrote the principles?
Does one company policy become a universal value across different laws and cultures?
Can people appeal?
Can users report biased behavior or mistaken refusals and see a correction process?
08 · Failure & Evaluation
Better instruction following alone does not prove greater safety.
When the objective is incomplete or evaluation is narrow, one score can improve while behavior worsens for another capability or population.
Reward gaming
Find superficial cues that raise an evaluation score without producing the intended outcome.
Sycophancy
Match a user’s belief rather than the facts because agreeable responses receive preference.
Over-refusal
Reject legitimate and beneficial questions while trying to reduce harmful output, or trigger on superficial wording.
Capability damage
Strong adaptation to a narrow domain or recent dataset weakens previous knowledge, languages, or reasoning.
Evaluation overfitting
Fit material similar to public tests so scores rise without carrying into unfamiliar reality.
Hidden behavioral shift
Follow rules in tests but act unexpectedly under different context, authority, or pressure.
Evaluate post-training at several layers.
- 01
Target task
Instruction following, correctness, format, and domain performance
- 02
Retained capability
Whether existing language, knowledge, and reasoning were damaged
- 03
Safety and fairness
Harmful output, bias, privacy, and excessive refusal
- 04
Adversarial tests
Euphemisms, long conversations, prompt attacks, and tool authority
- 05
Real deployment
Incidents, complaints, and new misuse across diverse users and environments
Questions for a published model-tuning report
- 01
Are the objectives and priorities of post-training stated concretely?
- 02
Can we inspect the source of demonstrations and preferences, rater composition, and rating guide?
- 03
Does the report isolate and compare the effect of each training stage?
- 04
Are results broken down by language, population, and high-risk domain rather than one average?
- 05
Were sycophancy, over-refusal, reward gaming, and damage to prior capabilities tested?
- 06
Are independent evaluation, incident reporting, correction, and rollback procedures available?
Check your understanding
Is post-training always the best way to add new facts?
No. It can adapt stable domain knowledge, but retrieval and provenance may be better for frequently changing facts. Separate the goal of behavior shaping from knowledge updating.
Is the response people prefer necessarily more truthful?
No. Raters may favor fluency, confidence, or agreement with their own views. Factuality needs independent evidence and testing.
Does DPO remove the need for human preferences?
No. DPO typically learns directly from chosen and rejected response pairs. It simplifies the separate reward-model and reinforcement-learning pipeline, but data-quality problems remain.
Does safety fine-tuning remove the need for deployment controls?
No. New attacks and contexts can appear, and refusal behavior can be bypassed. Permission limits, human approval, logs, monitoring, and incident response remain necessary.
09 · Study Path
Start with data for good answers, then study preference optimization and its failures.
You do not need to read every equation. Read the abstract, figures, and method in that order while repeatedly asking who defined a good answer and how.
Beginner · Complete pipeline
1–3 hoursInstructGPT
A representative pipeline connecting supervised demonstrations, human response rankings, reward modeling, and reinforcement learning.
Open learning resourceFLAN · Instruction Tuning
Research showing how fine-tuning across tasks expressed as natural-language instructions can improve instruction following on unseen task types.
Open learning resourceIntermediate · Compare preference methods
Days–weeksDirect Preference Optimization
The original DPO paper, directly adjusting the relative likelihood of chosen and rejected responses from preference pairs.
Open learning resourceConstitutional AI
An approach where a model critiques and revises responses from human-written principles and learns from AI-generated preference feedback.
Open learning resourceLoRA
The paper proposing low-rank trainable matrices for efficiently adapting a large model while keeping base weights frozen.
Open learning resourceAdvanced · Gaps in the objective
Weeks or moreSpecification Gaming
Examples of systems satisfying a measurable objective without achieving the outcome people intended, and the underlying reward-design problem.
Open learning resourceSycophancy in Language Models
Research into whether human preferences and reward models can encourage agreement with a user’s views over truthful answers.
Open learning resource