Deep Learning Guide · 02
How AI Performs Inference
In AI, ‘inference’ does not necessarily mean reasoning like a human. Training changes a model’s weights; inference is the execution stage that uses fixed weights and new input to compute an output.
The idea in three sentences
- The same model can produce different results depending on its input context, connected information and tools, and the method used to choose the next token.
- A language model does not retrieve a finished answer all at once; it repeatedly computes and selects one next token from the context so far.
- Fluent output is the result of computation, not proof of fact, so consequential claims require sources and independent verification.
01 · What Inference Means
Inference is the moment learned rules are applied to new input.
An image classifier computes category probabilities, a forecasting model produces a number, and a generative model makes new content. All are inference. Logical reasoning is one behavior that may appear during inference, not the definition of inference itself.
Changes the weights.
Repeated computation reduces errors across many examples. It often takes enormous compute and time, but a given model is trained a limited number of times.
Uses the weights.
Current weights compute an output for each new input. One run is smaller than training, but total compute, power, and cost can grow with users and output length.
Predictions can be computed in advance or on demand.
Static or batch inference
Recommendation lists or classifications are computed in bulk and cached. Results are fast to serve but cannot immediately reflect unusual new input.
Dynamic or online inference
A result is computed for each incoming request. It is flexible enough for conversation, but latency, peak load, and failure handling become important.
02 · Building the Input
What the model sees is larger than the sentence a user typed.
A conversational service assembles several pieces in a defined order. Hidden instructions and retrieved content can influence the output within the same context.
- 01
Service instructions
Roles, prohibited actions, and response formats set by the operator
- 02
Conversation history
Earlier user and AI messages used to interpret the current request
- 03
User input
Questions, commands, attached documents, or images in this request
- 04
External information
Search results, database passages, tool output, and retrieved memory
03 · One Token
Text becomes tokens, and the model scores every candidate for what comes next.
A single forward pass through a language model can be simplified as follows. A real Transformer performs this across many layers and attention heads in parallel.
- 01TokenizeConvert text into a sequence of token IDs
- 02EmbedRepresent tokens and positions as numeric vectors
- 03AttendCompute which parts of context should contribute
- 04TransformRepeatedly combine and transform representations
- 05LogitsAssign raw scores to every next-token candidate
- 06SelectTurn scores into probabilities and choose one candidate
04 · Autoregressive Loop
Choose one token, append it, and repeat to build an answer.
A decoder language model appends the selected token to the context and computes the next distribution again. The loop continues until an end token or length limit is reached.
Append the selected token ‘umbrella,’ then compute the next candidates again.
- 01Read context
- 02Score candidates
- 03Select next token
- 04Append to context
- 05Check for stop
05 · Decoding
The rule for choosing from the probability table is part of the result.
Without changing model weights, decoding settings alter stability, diversity, and length. No single setting is best for every task.
Greedy selection
Choose the highest-probability token at every step. It is reproducible but can become bland or repetitive over long output.
Temperature
Makes differences among candidate probabilities sharper or flatter. Lower is conservative; higher can diversify output while admitting less stable candidates.
Top-p
Build the smallest candidate set whose probabilities reach a threshold, then sample within it. The number of candidates changes with context.
Length and stop rules
Stop at a maximum number of new tokens or a defined string. Too short truncates answers; too long increases cost and accumulated error.
06 · Context & Memory
A context window is a workbench, not permanent memory or a store of truth.
The context window is the span of tokens a model can reference within one request. As a conversation grows, some content must be shortened or removed, and being inside the window does not mean every detail is used equally well.
Knowledge in weights
Information compressed into patterns during training. A prompt during inference does not directly rewrite it.
Current context
Instructions, conversation, and documents temporarily supplied for this computation. They do not remain in the model’s weights after the request.
External memory
Information a service stores in a database and inserts into a later request. Storage, deletion, and access are system policies outside the model.
KV cache
Intermediate attention values from processed tokens are kept to avoid repeating work for the next token. It is a computation cache, not a separate intelligence that remembers meaning.
07 · Retrieval & Tools
Retrieval and tools extend model knowledge—and bring risks from outside the model.
Basic inference produces output from weights and input. Real AI services connect search, code execution, APIs, and devices for current information and the ability to act.
- 01
Interpret request
The model or separate rules decide whether retrieval or a tool is needed
- 02
Query outside
Fetch candidates from a search index, database, or API
- 03
Select evidence
Insert relevant passages into context or construct tool arguments
- 04
Respond or act
Generate an answer or perform an allowed action using the result
Control points belong between model language and system action.
- Show original source and date with retrieved claims
- Validate tool arguments, scope, and permissions in code
- Require human approval for sending, deleting, paying, or other consequential actions
- Log results and failures, and provide a path to cancel or reverse
08 · Speed, Memory & Cost
Inference performance is measured in waiting and resources as well as answer quality.
Model size, input and output length, concurrent users, hardware, and optimization jointly determine perceived speed and operating cost.
- Time to first token
- Delay between a request and the first visible token. The cost of processing a long input strongly affects it.
- Tokens per second
- The rate of output after generation begins. It depends on model size, memory bandwidth, and other factors.
- Throughput
- Total requests or tokens a server handles over time. It is not always the same goal as the latency one person feels.
- Memory
- Space used by model weights and the KV cache. Longer contexts and concurrent requests increase cache pressure.
Batching
Compute several requests together to use hardware efficiently. Waiting to form larger batches can increase individual delay.
Quantization
Represent weights or cache in fewer bits to reduce memory and computation. Quality and speed tradeoffs must be evaluated.
Speculative decoding
A smaller assistant proposes tokens and a larger model verifies several at once, aiming to sample from the same distribution faster.
09 · Failure & Verification
Separate the fact that a model produced an answer from the claim that the answer is correct.
Inference failures can begin in the model, input context, decoding, retrieval, tools, or service policy. Consequential results need checks that are independent of generation.
Hallucination
Produce plausible context but false facts, nonexistent sources, or invented details.
Instruction conflict
Follow an unintended priority when service rules, user requests, and instructions inside documents conflict.
Context loss
Miss an important condition in long input or connect details from different places incorrectly.
Compounding error
An early mistaken assumption or token affects later generation and multiple stages of tool use.
Excess confidence
Use decisive language for probabilistic output, making uncertainty hard for a user to see.
Expanded authority
A wrong answer flows through automation into sending, deleting, purchasing, or physical action.
Five steps for checking an important answer
- 01
Separate claims from opinion and mark statements that need evidence
- 02
Open the primary source rather than trusting a linked title
- 03
Check whether date, jurisdiction, and population match the question
- 04
Cross-check calculations, code, and quotations with an independent tool or source
- 05
If being wrong causes serious harm, do not decide or act from AI output alone
Check your understanding
Do model weights change on every inference request?
Not in ordinary inference. Prompts and context change the current output, but changing weights requires a separate training process.
Does temperature near zero remove factual errors?
No. It reduces randomness but can repeatedly select the highest-probability candidate even when that candidate is wrong.
Is the context window the model’s long-term memory?
No. It is temporary working space for the current request. Persistent memory is usually stored outside the model and retrieved into a later request.
Does connecting search eliminate hallucination?
No. Search can return wrong or irrelevant material, and the model can misread or miscite it. The source and the final claim still need checking.
Study Path
Change generation settings yourself, then examine the system boundaries.
Begin with tokens and next-token selection. If you have development experience, continue into caching, retrieval, and tool execution.
Beginner · Concepts and experiments
45 minutes–3 hoursGoogle · Introduction to Large Language Models
An exercise-based introduction to tokens, context, self-attention, and how language models generate text.
Open learning resourceHugging Face · Generation strategies
Executable examples compare greedy decoding, sampling, and beam search as methods for selecting the next token.
Open learning resourceIntermediate · Connect model and system
Days–weeksAttention Is All You Need
The original Transformer paper. Its decoder masking and attention mechanisms form the technical basis of autoregressive generation.
Open learning resourceRetrieval-Augmented Generation
The original RAG paper combining learned parameters with external document retrieval for knowledge-intensive tasks.
Open learning resourceReAct
A paper proposing an agent pattern that interleaves language-model reasoning traces with actions and observations from an external environment.
Open learning resourceAdvanced · Output and efficiency
Weeks or moreThe Curious Case of Neural Text Degeneration
An analysis of repetition and blandness under maximum-likelihood decoding that introduced nucleus sampling.
Open learning resourceHugging Face · KV cache
A technical guide to reusing past attention values during autoregressive generation and the memory-speed tradeoff.
Open learning resource