Keywords: Large model Temperature, Top-p (kernel sampling), Softmax, sampling parameters, LLM parameter adjustment
Suitable for readers: developers who have called large model APIs but cannot figure out how to set temperature and top_p
Large model Temperature and Top-p are the two most common sampling parameters in the API and the most easily set incorrectly. Most articles on the Internet stop at the level of “the larger the value, the more random”, but this sentence neither explains why they are two parameters, nor can it guide you whether to fill in 0.3 or 0.7.
This article uses a set of real calculated probabilities to explain the role, mathematical mechanism, typical misunderstandings and parameter adjustment suggestions of the large model Temperature and Top-p at once.
1. Let’s first look at a set of numbers: What exactly has changed in Temperature?
Let me give the conclusion first:The large model Temperature and Top-p do not change “what to say” of the model, only “which one to choose from the candidates”.
Below are examples used throughout this article. The user asked “Recommend a technology channel that talks about AI for me.” The model scored the five candidate words (logits), and then converted them into probabilities through Softmax with a temperature coefficient:
sheet
| Candidate words | logit | T=0.1 | T=1.0 | T=2.0 |
|---|---|---|---|---|
| codigger | 3.0 | 66.5% | 28.2% | 24.2% |
| science and technology | 2.9 | 24.5% | 25.6% | 23.0% |
| AI | 2.8 | 9.0% | 23.1% | 21.9% |
| bike | 2.2 | ≈0% | 12.7% | 16.2% |
| waves | 2.0 | ≈0% | 10.4% | 14.7% |
Looking at the rows of numbers horizontally, the role of the large model Temperature is already clear: it does not add any new words, it just sharpens or flattens the original probability distribution. The numerical values are all actual calculations based on formulas, not schematic diagrams.
This also explains a common question: Why do answers become shorter and more templated after lowering the temperature? After the distribution becomes sharper, the model is more inclined to select those safe high-frequency words and connectives (“first”, “in general”), long sentences and uncommon expressions are systematically suppressed, so the text converges.
Two and three-step process: Which step does each of the two parameters act on?
Large models are generated by autoregression: only one word is output at a time, and then the word is inserted back into the input, repeating hundreds or thousands of times. A single output is divided into three steps:
- Logits: The model assigns an original score to each of the tens of thousands of candidate words in the vocabulary.
- Transition probability (Softmax): Convert the fraction into a probability distribution that sums to 100%.
- weighted sampling: Randomly select a word based on probability.
Temperature is used in step 2, and Top-p is used in step 3.
This is the most essential difference between the two, and it is also the starting point for understanding why they cannot replace each other. By the way, what is written in the API istemperatureandtop_pTwo fields, the values are 0–2 and 0–1 respectively.
2.1 Formulas and mechanisms
The complete Softmax has one more temperature coefficient T than the textbook version:
T appears in the denominator of the exponent, so it adjusts the relative difference between the fractions:
- The smaller T → the score is amplified → the advantage of the head word is exponentially strengthened → the distribution becomes sharper
- The larger T → the scores are compressed → all words tend to be average → the distribution becomes flat
Back to the example. When T=0.1, the gap between “codigger” and “technology”, which was originally only 0.1 points, is magnified ten times. After exponential amplification, it becomes 66.5% versus 24.5%, and the last three words add up to less than 10%. When T=2.0, “codigger” retreated from 28.2% to 24.2%, but “bicycle” increased from 12.7% to 16.2%.
Notice this asymmetry:Raising the temperature does far less damage to the head than it does to the tail.
This is the mathematical source of why the large model Temperature is “more creative” after it is increased – it is not that the model becomes smarter, but that it starts to be willing to choose words that are not usually qualified to be on stage.
2.2 Why is this parameter called “temperature”?
Physically, molecules are almost motionless at low temperatures and the structure is stable; thermal motion is violent at high temperatures and everything tends to be disordered. The large model Temperature borrows the same intuition: stable and conservative at low temperatures, and divergent and active at high temperatures. T→0 is ice and T→∞ is water vapor.
By the way, the value range is clear: the OpenAI interface accepts 0–2, Anthropic usually accepts 0–1, some Gemini models to 2, and domestic models such as DeepSeek and Tongyi mostly accept 0–2. If the range is exceeded, an error will be reported directly instead of automatically truncated.
3. Top-p core sampling: why Temperature alone is not enough
The large model Temperature has a structural flaw: it has only one curve, which acts on the entire vocabulary. You can’t just make “reasonable words” more and “outrageous words” less at the same time. The two share the same distribution.
The specific consequences are: under high temperatures, words such as “bicycle” that have nothing to do with the problem can get more than ten percentage points. If the number of samples is enough, one will be selected. A considerable part of the “this AI is talking nonsense” that users encounter in conversations comes from here.
Top-p (Nucleus Sampling) solves exactly this problem: it does not touch the probability value and directly cuts the candidate pool.
The rule is: accumulate from high to low according to probability. When the accumulation just reaches the threshold p, the door is closed and all remaining words are discarded.
Use the column T=1.0 in the table above to calculate the cumulative value:
sheet
| Ranking | Candidate words | Probability | cumulative probability |
|---|---|---|---|
| 1 | codigger | 28.2% | 28.2% |
| 2 | science and technology | 25.6% | 53.8% |
| 3 | AI | 23.1% | 76.9% |
| 4 | bike | 12.7% | 89.6% (reached 0.9) |
| 5 | waves | 10.4% | 100% |
When p=0.9, the first four words just make up 89.6%, and the fifth one is discarded; the remaining four words are put back to 100% in proportion and enter the sampling. If p is reduced to 0.5, only “codigger” and “technology” will remain, and even “AI” will not be considered.
3.1 The difference between Top-p and Top-k
sheet
| Comparative item | Top-k | Top-p (core sampling) |
|---|---|---|
| Filter by | Fixed number, always take the first K | Cumulative probability, when p is reached, it is truncated |
| Candidate pool size | fixed | Dynamically changes with distribution shape |
| When the distribution is very sharp | Still retain K words, mixed with low probability words | May only keep 1–2 |
| The distribution is very common | Insufficient candidates limit diversity | Automatically expand to dozens |
The candidate pool automatically scales with distribution, which is why Top-p replaces Top-k as the default scheme. Another point to add: OpenAI, Anthropic and other mainstream APIs no longer expose top_k to the outside world, only retaining temperature and top_p; locally deployed inference frameworks (vLLM, llama.cpp) usually support all three and can be freely combined.
4. The essential differences between the two and five common misunderstandings
sheet
| parameter | Action position | Change object | Turn down | Turn up | Typical misuse |
|---|---|---|---|---|---|
| Temperature | Softmax internal | Probability gap (steepness) | Winner takes all, output is almost fixed | The advantage of the head is reduced, and the long tail is on the table | Just adjust the temperature but complain about going off topic |
| Top-p | After softmax, before sampling | Candidate Pool Boundary | Only the head is left, stable but possibly rigid | The threshold is relaxed and noise comes in | I thought “0.2 means taking 20% of the words” |
Misunderstanding 1: Thinking that the higher the two parameters are, the more similar they will be.
“Turn them all up to be more random, and turn them all down to be more stable” is only true at the result level, and the mechanism is completely different. Increasing the Temperature of the large model will increase a 5% junk word to 15%; increasing the Top-p will only open the door wider and will not actively increase the original probability of any word. Temperature only amplifies the gap, not creates it – the gap is written in the scoring stage.
Misunderstanding 2: Thinking that T=0 is directly substituted into the formula
Mathematically, T=0 will divide by zero. In fact, the limit T→0+ is taken, and the distribution degenerates into one-hot. Engineering implementation usually treats it as a special case switch, using greedy decoding without entering random sampling; some frameworks do not really fall into greedy_search, but implement it by suppressing the probability to approximately one-hot, and the effect is close but not strictly equivalent.
Misunderstanding 3: Thinking that two parameter values can be copied across models
T acts on logits, and the logits scale of each model is inherently different – one model score often falls in 0-5, and another may fall in -20-20. For the same T=0.7, the sharpness of the distribution obtained is completely different. The parameters are only meaningful when you do comparative experiments on the link you call.
Misunderstanding 4: Thinking that the smaller the temperature setting is, the better
When the temperature approaches zero, decoding will degenerate into taking argmax every time, and the same prompt word will always have the same answer. This is an advantage in scenarios such as unit testing and JSON generation, but in scenarios where candidate diversity is required, it is a waste of sampling. What is more troublesome is that if the model has insufficient grasp of the problem, the tip distribution will only allow it to repeat errors more confidently.
Misunderstanding 5: Thinking that parameter adjustment can solve quality problems
These two parameters control output entropy, not output correctness. When the temperature is low, the model steadily outputs the “most likely answer” – and when it has no relevant knowledge, the most likely answer is often a smooth and confident nonsense. Certainty does not equal reliability, and randomness does not equal creativity.
5. Recommended configuration: fill in what value in what scenario
5.1 Value based on scenario
sheet
| scene | Temperature | Top-p | illustrate |
|---|---|---|---|
| Code generation, mathematical reasoning | 0.0–0.2 | 0.2–0.5 | Be deterministic and reduce context drift |
| Data extraction, translation, summarization | 0.2–0.4 | 0.4–0.7 | The structure is stable and allows minor modifications |
| Daily Q&A, emails, weekly reports | 0.7 | 0.9 | A common starting point, the most reliable |
| Brainstorming and naming | 0.9–1.2 | 0.9–0.95 | For diversity, multiple rounds of generation can be superimposed |
| Novels, scripts, copywriting | 1.0–1.5 | 0.9–0.95 | Temperature is responsible for “dare to speak”, p is responsible for “don’t be outrageous” |
5.2 Four practical principles
- Prioritize moving only Temperature. OpenAI officially recommends adjusting one of the two.
temperature=0.7matchtop_p=0.7It is often the worst combination – the diversity supported by high temperature is cut back by low p, leaving only incoherence. - Baseline set T=0.7, p=0.9. If the output is too floating, lower the T, if it is too stiff, raise the T; only when the output is still seriously off topic after lowering the T, move p.
- Write parameters into the configuration center. Don’t scatter them in the code, otherwise the same product will have three temperatures; T and p must also be written into the experimental record, otherwise group A will be 0.3 and group B will be 1.0. A considerable part of the difference in scores is just luck.
- Fix the prompt word first, then adjust the parameters. Treating these two parameters as “inspiration buttons” will often cause problems that should be positioned in retrieval quality or context length to be misjudged as “the model is not smart enough” – the cost of such misjudgements is usually to change models and suppliers.
6. Summary
Going back to the table at the beginning: the difference between 28.2% and 10.4%, in the final analysis, is that the model does not think about it and just repeatedly rolls the dice among tens of thousands of probabilities.
- large model Temperature Determine the eccentricity of the dice – change the steepness of the probability distribution by scaling logits;
- Top-p Determine how many sides the die has – truncate the candidate pool by cumulative probabilities.
Understand this division of labor, and you will be able to judge which one should be adjusted at what time: lowering T stably and raising T in a variety of ways; guarding against off-topic changes, and don’t copy across models.
