Temperature and Top-P: The 'Repeat' Tax

Most developers treat Temperature as a "Vibe" setting: 0.0 for math, 1.0 for creative writing. But Temperature is also a Financial Variable.

Higher temperatures increase the probability of "Loquacious" behavior. When a model is "Creative," it uses more adjectives, more adverbs, and more conversational filler. It is "Exploring" the token space rather than move toward the most probable (and usually shortest) answer.

In this lesson, we learn the Inference Economics of Randomness. We’ll explore how Temperature and Top-P (Nucleus Sampling) contribute to "Token Bloat" and how to tune them for "Maximum Density."

---

1. High Temperature = High Verbosity

At Temperature 0.0 (Greedy Decoding), the model picks the absolute most likely next token. This usually leads to direct, factual, and short responses.

At Temperature 1.0 (Creative), the model is allowed to pick less likely tokens.

• The Result: Instead of saying "Done," it says "The requested operation has been successfully completed and finalized for your records."
• The Cost: 15 tokens vs 1 token.

---

2. Top-P and the "Semantic Filter"

Top-P (Nucleus Sampling) limits the "Pool" of tokens the model can choose from.

• Top-P = 0.1: Only considers the top 10% of probability mass. (Very Focused).
• Top-P = 1.0: Considers all tokens. (Very Broad).

Efficiency Strategy: Use a low Top-P for Extraction and Logic. By "Pruning" the token pool early, you prevent the model from move into "Hallucinated Narratives" that waste tokens.

---

3. The "Repeat" Tax: Why Randomness adds turns

In an agentic loop (Module 9), high temperature is deadly.

1. Agent generates a "Creative" tool call.
2. The argument is slightly off (Hallucinated).
3. The Tool fails.
4. The Agent retries.

Total Cost: You paid for the "Creative" mistake AND the retry. By setting Temperature=0 for all agentic internal thoughts, you reduce the Retry Rate, which is the single biggest "Token Tax" in autonomous systems.

---

4. Implementation: Tuning the Request (Python)

Python Code: Task-Specific Inference Parameters

def call_optimized_llm(task_type, prompt):
    # DEFAULT: Maximum Efficiency (Cold)
    params = {"temp": 0.0, "top_p": 0.1}
    
    # EXCEPTION: Creative Writing
    if task_type == "creative_draft":
        params = {"temp": 0.8, "top_p": 0.9}
        
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=params['temp'],
        top_p=params['top_p']
    )

---

5. Frequency and Presence Penalties (The 'Anti-Bloat' Flags)

• Frequency Penalty: Penalizes tokens that have already appeared. (Prevents word loops).
• Presence Penalty: Penalizes any token that has appeared. (Forces new topics).

For token efficiency, you should use a mild Frequency Penalty (0.1 - 0.2). This prevents the model from getting stuck in a repetitive "Loop of Words" (e.g. "I will search... then I will search... then I will search...") that can consume thousands of tokens before an error is caught.

---

6. Summary and Key Takeaways

1. Cold is Cheap: Use Temp=0 for data, logic, and tool calls to ensure the shortest, most predictable path.
2. Top-P Pruning: Use low Top-P (0.1) for extraction to prevent "Narrative Drift."
3. Avoid Retry Loops: Randomness in agents leads to catastrophic retry costs.
4. Penalty Buffers: Use Frequency Penalties to kill "Infinite Token Loops" early.

In the next lesson, Max Tokens vs. Stop Sequences, we look at چگونه to put a "Hard Stop" on the model's generation.

---

Exercise: The Temperature Benchmark

1. Ask an LLM to "Describe a cat" 5 times.
2. Run 1: Temperature = 0.
3. Run 2: Temperature = 1.0.
4. Compare the lengths.

• Most students find that the Temp=1.0 responses vary wildly in length, but the Average Length is 20-30% higher than the Temp=0 version.
• Calculate: If you have 1 million "Descriptions," how many tokens did Temp=1.0 cost you in extra "Creative Fluff"?

---

Congratulations on completing Module 15 Lesson 1! You are now a master of inference parameters.