Core day 42 of 60 · Week 9 · Part II · orientation depth

LLM Architecture

A passenger chatbot is asked "Which gate does ZX104 leave from?" and answers "Gate A7", calmly and in flawless English. The gate system says B12. What exactly produced "A7", which settings could have changed the answer, and which could not?

By the end: You describe generation as a loop over next-token probabilities, state what temperature, top-k and top-p change and what they cannot, and explain why a fluent answer is not evidence that the answer is supported.

At a glance

A language model repeatedly scores every next token and samples one; decoding settings reshape that distribution but never add a missing fact.

Next-token loop
Score all tokens, sample one, append, repeat until a stop token.
Tokens
Subword pieces: an IBAN or a SKU may be several tokens; counts differ by language.
Temperature and top-p
Sharpen or flatten, then truncate the distribution; no new knowledge.
Context window
Prompt plus output must fit; a fitting prompt is not read evenly.
Fluent is not true
Supply facts through the prompt or a tool and verify identifiers in code.

Recall · 5 min

No reviews due? Answer from memory. (1) What did TF-IDF fail to do when a message used a word the labelled data never contained (Day 41)? (2) In self-attention, what do queries and keys decide (Day 39)? (3) Why does a softmax output sum to one, and what does dividing the inputs by a number above one do to it?

Open due reviews

Concept · 10 min

A language model does one thing: given the Tokens so far, it scores every possible next token. A sampler picks one, appends it and the loop repeats. "Gate A7" appears because gate-like tokens are likely after "departs from gate", not because anything was looked up, so a wrong answer reads exactly as fluently as a right one.

Temperature and top-p change which token is drawn from a fixed distribution; they never change what the model knows

Predict first

In the invented-paper prompt you lower the temperature from 1.0 to 0.2. What happens to the share of draws that are an invented name?

Text description of the figure

A bar chart of a fixed, hand-written toy distribution over candidate next tokens (not a real model), with two prompts: a capital-city fact where the supported answer is the most likely token, and an invented paper title with no supported author. Controls set temperature, top-k, top-p, the number of seeded draws and the seed. Outline bars show the base probabilities and filled bars the reshaped ones; readouts give the probability of the supported answer, of fluent but unsupported tokens and of hedges, plus entropy and the drawn tallies. The gate example appears only in the application code.

Four ideas to hold on to Overview

Tokens in
Text becomes integer ids from a fixed subword vocabulary.
Scores out
One logit per vocabulary token at the last position.
Sampling reshapes
Temperature and top-p change which token is drawn, not what is known.
No lookup
Nothing in the loop checks a fact; verification lives outside the model.

Generation is one loop: score every token, reshape, draw one, append Loop

Tokens so farprompt ids, then every generated idforward pass ↓
Transformer blocksfixed weights, masked self-attentionlast position ↓
Logitsone score per vocabulary tokendivide by T ↓
Softmax at T, top-psharpen or flatten, then cut the taildraw ↓
One tokensampled, or the top one if greedyappend ↓
↺ back to 1 · Tokens so far

Steps 1 to 3 are the model, with weights fixed after pretraining. Steps 4 and 5 are the decoder: it only reshapes and samples the distribution the model produced. Nothing in the loop looks a fact up.

The generation loop, one token at a time Illustration

Gate tokens get the share each gate has after "gate" in the training text; A7 wins with 0.33. Flight ZX104 plays no part.

"at" is appended (0.65) and the model is asked again with the longer text.

A near-tie: greedy takes 14:30 (0.20) over 10:15 (0.18). Sampling would often pick another, equally fluent value.

After "." the stop token has probability 1.00, so the loop ends.

Smallest possible language model: bigram counts from 400 seeded synthetic announcements. Greedy decoding of "flight zx104 departs from gate" gives "flight zx104 departs from gate a7 at 14:30 ." (the true gate in this story is B12). A real model conditions on the whole context, but it also emits the likeliest continuation, not a looked-up fact.

A bigram model counted from synthetic announcements, decoded greedily. Each step scores every next token, takes one and appends it.

Temperature and top-p as small multiples: same scores, six settings Illustration

No gate in context. Entropy 0.79 bits; P(B12) 0.04; top-p 0.8 keeps 1 token, P(B12) after it 0.00.

No gate in context. Entropy 2.23 bits; P(B12) 0.20; top-p 0.8 keeps 4 tokens, P(B12) after it 0.22.

No gate in context. Entropy 2.47 bits; P(B12) 0.19; top-p 0.8 keeps 4 tokens, P(B12) after it 0.24.

Gate B12 in context. Entropy 0.00 bits; P(B12) 1.00; top-p 0.8 keeps 1 token, P(B12) after it 1.00.

Gate B12 in context. Entropy 0.97 bits; P(B12) 0.83; top-p 0.8 keeps 1 token, P(B12) after it 1.00.

Gate B12 in context. Entropy 2.05 bits; P(B12) 0.52; top-p 0.8 keeps 3 tokens, P(B12) after it 0.65.

A7, the model’s favourite (wrong)
B12, the gate in the gate system
hollow: cut by top-p 0.8

Hand-set toy scores from the application code (not a real model); probabilities computed exactly with the next-token-sampler widget’s code.

Top row without the gate in context, bottom row with it. Lower temperature sharpens whatever the model already favours.

Where a fact can come from Comparison

Criterion	Weightslearned in pretraining	Prompttext you place in context	Tool callmodel requests a lookup
Freshness	Frozen at a training cutoff	As fresh as the text you pass	Live
Private or live data	Never: an account balance or parcel status is in no training set	Yes, if you fetch and insert it	Yes, through an allowed function
Can be checked	No source to point to	Yes: the answer can cite the line	Yes: compare output with the tool result
Cost	Free at inference	Tokens, latency, window space	A round trip and a contract to maintain (Day 47)

Loop: tokenize -> embed -> masked transformer blocks (Day 39) -> logits -> Temperature (sampling) and top-p -> sample -> append.
The weights are fixed after Pretraining; generation is inference only, and decoding settings only reshape the computed distribution.
The Context window holds prompt plus output; a KV cache keeps earlier keys and values and grows with length.
An answer supported by neither context nor world is a Hallucination; the fix is evidence (prompt, tool) plus verification.

Assumptions

The description fits decoder-only models in general; real products add system prompts, tools and post-training that change what is sampled.
The toy distribution below is hand-set and tiny; real vocabularies have tens of thousands of tokens.
Probability is not truthfulness. It reflects patterns in training text and the prompt, not a check against reality.

Application and workplace decision · 10 min

A toy sampler stands in for the chatbot's last step. Scores after "Flight ZX104 departs from gate" are hand-set (toy, not a real model): A7 is most likely, the true gate B12 is third.
A second score vector adds "ZX104: gate B12" to the context, which raises B12.
The code draws 5,000 samples for four settings with and without context.
Predict first: without context, which setting gives the most correct answers, and does any make the bot reliable?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

tokens = np.array(["A7", "C3", "B12", "D9", "the", "unknown"])  # toy candidates after "Flight ZX104 departs from gate"
no_context = np.array([2.6, 2.2, 2.0, 1.5, 0.5, 0.3])  # hand-set toy scores, not from a real model
with_context = np.array([2.6, 2.2, 5.0, 1.5, 0.5, 0.3])  # same prompt plus "ZX104: gate B12" in the context
truth = "B12"  # the value in the gate system

def next_token_probs(logits, temperature=1.0, top_p=1.0):
    p = np.exp(logits / temperature)
    p /= p.sum()
    order = np.argsort(-p)
    n = min(np.searchsorted(np.cumsum(p[order]), top_p) + 1, len(p))  # smallest set reaching mass top_p
    q = np.zeros_like(p)
    q[order[:n]] = p[order[:n]]
    return q / q.sum(), n

rng = np.random.default_rng(42)
for label, logits in [("no context", no_context), ("with context", with_context)]:
    for t, tp in [(0.2, 1.0), (1.0, 1.0), (1.0, 0.8), (2.0, 1.0)]:
        q, n = next_token_probs(logits, t, tp)
        draws = rng.choice(tokens, size=5000, p=q)
        h = abs(-(q[q > 0] * np.log2(q[q > 0])).sum())
        print(f"{label:<12} T={t:<3} top_p={tp:<3} kept={n} entropy={h:.2f} bits  "
              f"P(top)={q.max():.2f} top={tokens[q.argmax()]:<3} correct={np.mean(draws == truth):.2f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The model, defined, and what is inside it4 minThe autoregressive factorisation every decoder-only model computes, the stack from embeddings to logits, and a rough parameter count.

The model, defined: a product of next-token distributions Formula

p_\theta(x_1,\dots,x_N) = \prod_{t=1}^{N} p_\theta(x_t \mid x_{<t}), \qquad p_\theta(\cdot \mid x_{<t}) = \mathrm{softmax}(W_U\, h_t)

$x_t$: token id at position t, from a fixed vocabulary of size V
$p_\theta(x_t \mid x_{<t})$: the distribution the network outputs at step t, given only earlier tokens (the causal mask, Day 39)
$h_t$: the last block's vector at position t, d numbers
$W_U$: unembedding matrix, V x d: one row per vocabulary token, so W_U h_t is the vector of V logits
$\prod$: the chain rule of probability: exact for any sequence, no independence assumption; the network only parameterises each factor

Generation samples the factors one at a time, left to right. Training (Day 43) maximises the same product on real text. Nothing in either step consults a fact base.

Inside a decoder-only model, bottom to top Stack

Token embedding + positionEach id becomes a vector; position information is added or rotated in.
Masked self-attentionEach position reads only earlier positions (Day 39).
Feed-forward networkA per-position MLP, classically 4 times wider than the model; gated variants differ.
Residual + normalisationAround each sub-layer; repeated for L blocks.
UnembeddingThe last position's vector times the vocabulary matrix gives the logits.

Rough parameter count of a decoder stack Formula

N \approx 12\, L\, d^{2} + V d

$12\, L\, d^{2}$: L blocks, each about 4d^2 attention plus 8d^2 feed-forward weights (width 4d)
$L$: number of blocks (layers)
$d$: model width
$V d$: embedding table, V = vocabulary size

L=32, d=4,096 gives about 6.4 billion block weights: an order-of-magnitude check, not a spec; gated MLPs, grouped-query attention and mixtures of experts change it. Each generated token costs about 2N floating-point operations (one multiply and one add per weight; Kaplan 2020), so decoding cost scales with model size times answer length.

Sampling, derived and computed by hand5 minThe temperature formula pushed through four logits on paper, top-p on the same numbers, and the four common decoders.

Temperature-scaled softmax Formula

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

$z_i$: logit of token i at the last position
$T < 1$: sharpens; T approaching 0 is greedy decoding
$T > 1$: flattens; unlikely tokens are drawn more often

The ordering of tokens never changes with T; only how much mass the top ones get.

Four logits through temperature, by hand (delivery-chat toy, synthetic) Table

Token after "Your order will arrive"	Logit z	exp(z), T=1	p at T=1	p at T=0.5	p at T=2
tomorrow	2.0	7.389	0.61	0.84	0.43
Friday	1.0	2.718	0.22	0.11	0.26
soon	0.5	1.649	0.14	0.04	0.21
never	-1.0	0.368	0.03	0.002	0.10
sum (divide by it)	-	12.124	1.00	1.00	1.00

Hand-set logits for an e-commerce delivery chat, not a real model. Divide z by T, exponentiate, divide by the sum; checked in numpy. Cooling to T=0.5 moves "tomorrow" from 0.61 to 0.84, real date or not.

Top-p 0.9 on the same four tokens, by hand Sequence

Sort
At T=1 the sorted probabilities are 0.61, 0.22, 0.14, 0.03.
Accumulate
Cumulative mass: 0.61, then 0.83, then 0.97: the first value at or above 0.9.
Truncate
Keep the smallest prefix reaching 0.9: three tokens; 'never' (0.03) is cut.
Renormalise
Divide the kept 0.61, 0.22, 0.14 by 0.97: final 0.63, 0.23, 0.14.
Read it
Top-p deleted only the tail. The favourite is unchanged; no new token can ever enter.

Same toy logits as the worked table, checked in numpy. Top-k would cut at a fixed count instead of a fixed mass.

Decoding strategies Table

Strategy	What it does	Typical use	Watch out for
Greedy	Always take the top token	Extraction, classification	Repetition on long text
Temperature	Rescale logits before softmax	Low for facts, higher for drafts	Not a truth dial
Top-k (Fan 2018)	Keep the k most likely tokens	Cap the tail	Fixed k ignores how peaked p is
Top-p (Holtzman 2020)	Keep the smallest set with mass p	Drafting, around 0.9-0.95 by convention	Values are conventions to test

Decoding defaults in Hugging Face Transformers (as read 2026-10-12) Table

Setting	Documented default	Expert note
do_sample	off: greedy decoding	Temperature and top-p do nothing until sampling is on
temperature	1.0 if not set	A model's generation_config.json can override it
top_k	50 if not set	Silently truncates the tail when sampling; set it explicitly
top_p	1.0 if not set (no truncation)	Applied after top-k when both are set

Hosted APIs use their own defaults and parameter names. Record the exact settings with every evaluation run.

Properties of decoding: what it can and cannot change4 minRanking, log-odds, entropy and the two limits of temperature, each checked on the worked logits, then the same facts on the application's gate scores.

Properties of temperature, each checked on the four delivery-chat logits Table

Property	Why it holds	On the worked logits
Ranking never changes	z/T keeps the order of z for any T > 0, and exp is increasing	tomorrow > Friday > soon > never at T = 0.5, 1 and 2
Log-odds scale by 1/T	log(p_i / p_j) = (z_i - z_j) / T	tomorrow vs Friday: odds e^1 = 2.72 at T = 1, e^2 = 7.39 at T = 0.5
Entropy rises with T	Standard result for softmax(z/T) when the logits are not all equal	0.78 bits at T = 0.5, 1.46 at T = 1, 1.82 at T = 2
T to 0 is greedy	All mass moves to the largest logit	p(tomorrow) 0.61, then 0.84, then 1 in the limit
T to infinity is uniform	z/T goes to 0 for every token	2.00 bits = log2(4) in the limit, whatever the logits said
Truncation only deletes	Top-k and top-p zero the tail and renormalise	No setting moves a token up the ranking; a cut token gets exactly 0

Entropy values computed in numpy. The first and last rows are the lesson in two lines: decoding re-weights the model's own ranking and never adds an option it did not score.

Lower temperature makes the top token more certain, right or wrong (toy scores) Chart

A7, wrong gate, no context
B12, true gate, no context
B12 with the gate in context

Exact softmax probabilities of the application's hand-set scores. Without evidence, cooling the sampler makes the wrong gate near-certain; with evidence, it makes the right gate certain.

Share of correct gates in 5,000 draws (application output) Chart

No context, T=0.20.04 share correct
No context, T=1.00.2 share correct
No context, top-p 0.80.22 share correct
No context, T=2.00.2 share correct
Context, T=0.21 share correct
Context, T=1.00.84 share correct
Context, top-p 0.81 share correct
Context, T=2.00.52 share correct

No setting rescues the no-context case. Once the evidence is in context, low temperature and truncation remove the residual wrong draws.

Tokens and the vocabulary3 minHow byte-pair encoding builds the vocabulary and why token counts differ by language and by identifier.

How a subword vocabulary is built (byte-pair encoding) Sequence

Start small
Begin with single characters or bytes as the vocabulary.
Count pairs
Count adjacent symbol pairs across the training text.
Merge
Add the most frequent pair as a new symbol.
Repeat
Stop at the target vocabulary size.
Result
Common words are one token; rare words and codes split into pieces.
Consequence
An IBAN, SKU or ICD-10 code becomes several tokens, each a separate guess: verify identifiers in code.

Sennrich et al. (2016). Budgets, prices and window limits are counted in tokens, so count with the deployed model's own tokenizer: text from languages that were rare in its corpus needs more tokens for the same content (Petrov et al. 2023).

Budgeting the context window3 minWhat shares the window and why a prompt that fits can still be read unevenly.

A context budget for an insurance claims assistant (hypothetical) Chart

System prompt
Retrieved passages
Tool output
User question
Maximum answeroutput shares the window
Window

Illustrative values: they show the pattern, not measured results.

Everything left of the window bar is summed against it (7,800 of 8,192 here). Count with the deployed model's own tokenizer.

KV-cache arithmeticadvanced3 minMemory per token and per sequence, and why long contexts reduce serving throughput.

KV-cache memory per token Formula

M_{token} = 2 \cdot L \cdot n_{kv} \cdot d_{head} \cdot b

$2$: one key and one value
$L$: number of layers
$n_{kv}$: key-value heads (fewer than query heads with grouped-query attention)
$d_{head}$: dimension per head
$b$: bytes per number (2 for 16-bit)

KV cache by hand: 128 KiB per token becomes 100 GiB per server Sequence

Per token
2 x 32 layers x 8 KV heads x 128 dims x 2 bytes = 131,072 bytes = 128 KiB.
Per sequence
Linear in length: 32,768 tokens x 128 KiB = 4 GiB of cache, before weights and activations.
Per server
Times concurrency: 100 requests of 8,192 tokens = 100 x 1 GiB = 100 GiB. This is what caps batch size.
Fewer KV heads
Grouped-query attention shares each KV head among several query heads (Ainslie 2023): 8 instead of 32 cuts it 4x.
Fewer bits
An 8-bit cache halves it again (64 KiB per token); test accuracy at your own lengths before relying on it.

Hypothetical configuration (32 layers, 32 query heads of 128 dims, 16-bit numbers), checked in Python.

KV cache against context length, and what it does to batch size Illustration

One sequence. Memory grows linearly with length: 128 KiB per token with 8 KV heads, so 4 GiB at 32k tokens. With 32 KV heads a single 32k request needs more than the 12.0 GiB of weights.

Concurrent sequences that fit in the 68 GiB left on a 80 GiB card after the weights. With 8 KV heads: 136 at 4k tokens, 17 at 32k, 4 at 128k. Long contexts trade away batch size.

32 KV heads (multi-head): 512 KiB/token
8 KV heads (grouped-query): 128 KiB/token
1 KV head (multi-query): 16 KiB/token

Arithmetic on a hypothetical configuration (32 layers, head dimension 128, 16-bit cache, 6.4e9 block weights, an 80 GiB card); activations, fragmentation and framework overhead are ignored, so real capacity is lower.

Linear growth per sequence; fewer key-value heads (grouped-query, multi-query attention) cut it by the head ratio.

The two phases of serving a request Comparison

Criterion	Prefillread the prompt	Decodewrite the answer
What runs	All prompt tokens in parallel through every layer	One token per step, reusing the KV cache
Grows with	Prompt length (attention part grows quadratically)	Answer length times model size
Usually limited by	Compute	Memory bandwidth: weights and cache are read every step
Lever	Shorter prompts; caching a shared prefix	Batching requests; smaller cache per token

A practitioner model of serving cost, consistent with the paged-attention paper's description of the cache (Kwon et al. 2023). Measure latency per phase.

Diagnosing hallucination3 minA decision path from a wrong, fluent answer to the control that fixes it.

Diagnosing a wrong, fluent answer Diagram

Wrong fluent answer
Fact in context or tool?
- no: to Supply it
yes
Was it used?position, length, conflicts
- no: to Shorten or reorder
Supply itprompt, retrieval or tool
- then: to Verify in code
Shorten or reorderthe context; test at real lengths
then
Verify in codeagainst the system of record

Both branches end at a check in code: no setting inside the loop verifies a fact. Why models guess at all is evolving research; Kalai et al. (2025, preprint) argue that training and scoring reward guessing over abstaining.

Four ways the token loop misleads Overview

Confident wrong facts
Plausible tokens without support; same tone as right ones.
Stale knowledge
Weights stop at a cutoff; prices, tariffs and policies move on.
Lost in a long prompt
Fitting the window does not mean every clause is used.
Repetition
Greedy decoding on long text loops (Holtzman 2020).

Interview nuances and common misreadingsadvanced2 minFive statements that sound right in an interview and what a staff-level answer says instead.

Common misreadings, and the staff-level correction Overview

Temperature 0 means truth
It means the top token every time; hosted systems may not even be bitwise repeatable.
Probability is confidence
Token probabilities are uncalibrated for facts until checked on labelled data.
Window size is memory
The window is per request; nothing persists between calls unless you resend it.
Longer context is free
It costs prefill compute, KV memory and batch size, and may be used unevenly.
Tokens are words
Codes, numbers and rarer languages take more tokens per word.

Memory anchor

A model emits likely tokens, not verified facts: fix support with context, then verify.

Useful for

Explaining to stakeholders why a confident answer needs a source, and which control (context, tool, verification) fixes what.
Budgeting prompts: counting prompt plus output tokens, and expecting memory and latency to grow with context length.
Choosing decoding settings per task: low temperature for extraction and classification, higher for drafting variants.

Not suitable for

Predicting how a specific hosted model behaves; products differ in tokenizer, post-training, system prompt and sampling defaults, so measure on your own cases.
Treating token probabilities as calibrated confidence in a fact without a calibration check on labelled data (Day 45).

Traps

Using temperature as a truth dial

A bank's card-support team lowers temperature to cut invented fee amounts. The answers become repeatable, including the wrong ones, and the change is credited with a safety it does not provide.

How to avoid it

Treat temperature as a variance control. Fix unsupported answers with context, retrieval or tools, and verify identifiers and numbers against a system of record.

Counting only the prompt against the window

A telecom contract-summary bot passes a 7,500-token contract to an 8,192-token model. Output tokens, system prompts and tool results share the same window, so the request fails or the summary is cut mid-sentence.

How to avoid it

Budget prompt, retrieved text, tool output and maximum answer length together, and count tokens with the deployed model's own tokenizer.

Senior reasoning

A senior separates what the model computed from what the product guaranteed. They ask where the fact came from: the weights (frozen at a cutoff), the prompt, or a tool. They pick decoding settings per task, never to cure unsupported output, budget the window and cache, and put a verifier between the model and the customer for identifiers, amounts and dates: an account number in banking, a dosage in a clinic's patient line, a delivery date in logistics. They also test at the lengths and languages of real traffic rather than demo prompts.

Interview question

A hospital's appointment-line assistant gave a patient the wrong clinic opening hours, and a stakeholder says "the model hallucinated, can we just turn the temperature down?" How do you respond?

Outline your answer first.

Recall cards scheduled after completion

What do temperature, top-k and top-p change, and what can they not change?
What does the context window have to hold, and what is the KV cache for?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017core
The transformer with masked self-attention and positions that decoder-only language models build on.
Language Models are Unsupervised Multitask Learners — Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019optional
A decoder-only language model trained on next-token prediction and used for text generation without task-specific training.
The Curious Case of Neural Text Degeneration — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi, 2020core
Explains why maximisation decoding produces repetitive text and defines nucleus (top-p) sampling.
Hierarchical Neural Story Generation — Angela Fan, Mike Lewis, Yann Dauphin, 2018optional
Early use of top-k sampling and a tuned softmax temperature for generation (Section 5.4). (Section 5.4, Generation)
Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016optional
Byte-pair-encoding subword units that make open-vocabulary tokenization practical.
Language Model Tokenizers Introduce Unfairness Between Languages — Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, Adel Bibi, 2023optional
The same text can need many times more tokens in some languages than in others, so budgets must be measured per language.
Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020optional
Pretraining loss falls as a power law in model size, data and compute; says nothing about any single fact.
RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024optional
Evidence that advertised context lengths are not the lengths at which models are reliably usable.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai, 2023optional
Grouped-query attention, which shares key-value heads among query heads and shrinks the KV cache; used in the KV-cache chapter.
Transformers documentation — Hugging Faceoptional
GenerationConfig documents the decoding defaults (greedy unless sampling is on; temperature 1.0, top_k 50, top_p 1.0 when unset), as read 2026-10-12.
Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023optional
Describes the per-request key-value cache as large and dynamically growing, limiting batch size and throughput.
Why Language Models Hallucinate — Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang, 2025optional
Argues that training and evaluation reward guessing over abstaining, so hallucination persists; a preprint, evolving research.
Survey of Hallucination in Natural Language Generation — Ziwei Ji et al., 2023optional
Taxonomy of intrinsic and extrinsic hallucination in generated text.

Optional sections

Deep dive · 10 min

Parameter count, roughly. For a standard decoder block with model width $d$ and a feed-forward width of $4d$ , attention holds about $4d^2$ weights and the feed-forward network about $8d^2$ , so $L$ blocks hold about $12\,L\,d^2$ . A hypothetical configuration with $L=32$ and $d=4096$ gives about 6.4 billion weights, plus the embedding table ( $V\cdot d$ ). Real models vary (gated feed-forward layers, grouped-query attention, mixture of experts), so treat this as an order-of-magnitude check, not a spec.

KV cache arithmetic (approximate, hypothetical configuration). Per token the cache stores keys and values for every layer: $2\times L\times n_{kv}\times d_{head}\times$ bytes. With $L=32$ , 8 key-value heads, $d_{head}=128$ and 2-byte numbers, that is 131,072 bytes (128 KiB) per token, so a single 32,768-token sequence needs about 4 GiB before weights and activations. Serving many requests multiplies it, which is why long contexts reduce batch size.

What pretraining optimises. Cross-entropy on the next token over a very large corpus. Loss falls smoothly with model size, data and compute (Kaplan 2020), but low average loss says nothing about any single fact. Rare facts tend to fail first (insight-0504), and facts after the training cutoff are absent.

Sampling in practice. Greedy decoding is deterministic given identical numerics but tends to repeat itself on long text. Sampling with temperature around 0.7-1.0 and top-p around 0.9-0.95 is a common drafting setting; these values are conventions to test, not constants. For extraction, classification and tool calls, use low temperature and validate the output format (Day 47).

Retrieval project link (weekend studio). In the Sunday recall block of the retrieval milestone you are asked why a fluent answer is not evidence and what temperature and top-p change. Use today's script as the worked example: the answer is that they reshape a distribution, so the evidence has to come from the retrieved chunk.

Interview practice · 10 min

A logistics company's chatbot states a wrong delivery date in a confident, polished sentence. Explain to the product owner, without maths, why that happens. Then say what you would build so it happens less.

Answer as you would in the interview.

Content version 7 · last verified 2026-10-18 · depends on numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 27 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min