Core LLM Elements

Large language models come with a set of controls that shape how they behave: how creative or precise they are, how much they can remember within a conversation, and how they decide what to say next. These controls are not complicated, but they do build on each other. This guide walks through the core elements one by one: what they are, how they work, and how they connect.

Tokens: The Building Blocks of Language

Tokens are the smallest units of language that an LLM can process. For humans, the smallest unit of language is a single character. But for LLMs, a token can represent a character, part of a word, a whole word, or even a short phrase.

For example, take the word amoral. The "a" at the beginning is a prefix that changes the meaning of the word "moral." So "amoral" might be split into two tokens: a and moral. In contrast, the word "cat" is a single token because the letters do not carry independent meaning.

A good rule of thumb is that an average English word is about 1.5 tokens. So, if you write 100 words, you’re likely generating around 150 tokens. The tool that converts text into tokens is called a tokenizer. Different tokenizers might split the same text into tokens differently, but the general principle remains the same.

This matters for two reasons you will care about immediately:

• Cost: every API call is billed by the number of tokens, both what you send in and what the model sends back.
• Limits: every model has a maximum number of tokens it can handle in one go. Once you exceed that limit, something gets cut off.

One more thing worth knowing: tokenization behaves very differently across languages. The same sentence in Arabic will use significantly more tokens than in English, because most models are trained predominantly on English text and their tokenizers are optimized for it. This is directly relevant if you are building bilingual products for the region.

Context Window: The Model's Working Memory

A context window represents the “working memory” of an LLM. It determines how much information from a conversation or input the model can retain and use to generate subsequent responses. When you send a message, the model does not just read your new message in isolation, it reads the entire conversation from the beginning, your messages, its responses, any instructions you gave it upfront, all of it, every single time. All of that has to fit inside the context window.

Imagine there is a box that represents the other person's working memory. If the entire conversation fits inside that box, they can remember everything you have said and respond appropriately. But if the conversation grows too long and overflows the box, they start to forget the earlier parts.

The context window is measured in tokens, not words. This means that a long conversation can fill up the context window quickly, even if it does not seem like a lot of text. For example, a conversation with 10,000 tokens would fill up a context window of 8,000 tokens, leaving no room for the model to generate a response.

Context windows have grown significantly over time. Early LLMs had context windows of around 2,000 tokens. Today, many models handle 128,000 tokens or more. At first glance, 128,000 tokens might seem like overkill - after all, how often do you need to process a conversation or document that long? But context windows aren’t just for conversations. They also store:

A few long documents or snippets of code can quickly fill up even a large context window.

Understanding context windows is crucial if you want to get the most out of LLMs. Whether you’re building a chatbot, analyzing documents, or generating code, you need to know how much information the model can handle at once.

So, the next time you interact with an LLM, think about the context window. It’s the invisible box that determines how much the model can remember, and how well it can respond.

Sampling Parameters: How the Model Chooses the Next Word

Here is the connection. A Transformer processes your entire input - every token you sent - through layers of attention and feed-forward operations. At the very end of that process, it outputs a single thing: a score for every token in its vocabulary, indicating how likely each one is to come next. Those raw scores are called logits. A function called softmax then converts them into probabilities that sum to 1.

So when an LLM generates a response, it is not "knowing" the answer the way a human does. At every single step, it produces a probability distribution over its entire vocabulary, often 100,000+ possible next tokens. Given the prompt "The sky is…", the model might calculate that "blue" has an 80% probability, "cloudy" has 15%, "green" has 0.1%, and so on. It picks one token based on those odds, adds it to the sequence, runs the whole Transformer forward pass again, and repeats until the response is done.

Once you understand that the model is picking from a probability distribution at every step, the next question is: what controls that picking? That is exactly what sampling parameters are. They are the settings you configure - either through an API call or a model playground - to shape how the model behaves when it generates output. They do not change the model's knowledge or training. They change the rules for how it samples from the distribution it already computed.

Every sampling parameter ultimately answers one of four practical questions:

Temperature

Temperature is the most important sampling parameter you will work with. It controls the randomness of the model's output by scaling the logits before the softmax function converts them into probabilities.

A lower temperature sharpens the distribution, the top token gets most of the probability mass and the model behaves predictably. A higher temperature flattens it, lower-probability tokens get more of a chance and the output becomes more varied and creative.

The scale runs from 0 to 2 in most APIs, with a default of 1. Here is what each range actually does and when to use it:

Temperature range reference

In practice, you will rarely need to go above 1.2. Start at 0.7 for most tasks and adjust from there, lower if the output feels too loose, higher if it feels too repetitive or formulaic.

Top-K and Top-P

Temperature reshapes the probability distribution. Top-K and Top-P are filters that cut it down before sampling even begins, preventing the model from ever picking genuinely unlikely or nonsensical tokens, regardless of what temperature says.

Top-K (Hard Token Cutoff)

Top-K is the simpler of the two. Setting top_k = 4 means the model only considers the four most probable tokens. Everything else is discarded before sampling. An additional softmax can be applied to the surviving tokens so their probabilities sum to 1.

Top-K quick reference

Top-P (Nucleus Sampling)

Top-P works differently from Top-K. Instead of fixing a hard number of tokens to keep, you set a probability threshold between 0 and 1. The model then walks down its sorted list of tokens - starting from the most probable - and keeps adding them to the candidate pool until their combined probability reaches that threshold. Everything after that point is discarded.

For example, setting top_p = 0.8 means: "keep adding the most likely tokens until together they account for 80% of the probability mass, then stop." If the model is very confident about the next word, just 2 or 3 tokens might already cover 80%, so the pool stays small. If the model is uncertain and probability is spread across many tokens, it might need 30 or 40 tokens to reach 80%, so the pool grows automatically. This adaptive behaviour is exactly what makes Top-P more flexible than Top-K.

In practice, Top-P is generally preferred over Top-K because it adapts to the model's own level of confidence at each step. Most APIs use it as their default.

Top-P quick reference

Temperature vs Top-P vs Top-K: Head-to-Head

This is one of the most commonly asked questions about LLM sampling parameters. Here is a side-by-side comparison to make the distinctions concrete:

Max Tokens

What It Does

Max tokens (also max_tokens, max_output_tokens, or max_completion_tokens depending on the API) sets the maximum number of tokens the model will generate in its response. It is a hard upper limit - the model may stop earlier at a natural ending or stop sequence, but it will never exceed this count.

Important Distinctions by Provider (April 2026)

Repetition Penalties

Left to itself, a model will sometimes repeat itself, the same word three times in a paragraph, the same phrase it already used, or in extreme cases the same sentence indefinitely. This happens because once a pattern appears in the output, the model's probability distribution makes that pattern statistically attractive again.

Repetition penalties fix this by reducing the probability of tokens the model has already generated. There are two variants you will encounter across different APIs:

The frequency penalty specifically targets tokens that are used repeatedly, lowering their score each time they appear. The more often a token is used, the less likely the model is to use it again. The presence penalty applies immediately - as soon as a token has appeared at all, it is penalized, pushing the model toward introducing new vocabulary and new ideas.

In practice, you will reach for the frequency penalty when your model starts echoing itself, the same word appearing three times in a paragraph, or a phrase used earlier creeping back in. It is most noticeable in long outputs like reports or summaries. A small value around 0.3 is usually enough to clean this up. You rarely need to go higher.

The presence penalty solves a different problem. Use it when the model keeps circling the same idea without moving forward, rephrasing the same point in slightly different words instead of covering new ground. It is useful for creative tasks, brainstorming, or any situation where you want the output to explore rather than consolidate.

Most of the time you will not need either. When you do, start low, adjust one at a time, and stop as soon as the output feels natural again.

Putting It All Together

These parameters do not work in isolation. Every time the model generates a single token, they run in order:

Everything covered in this guide: tokens, context windows, sampling parameters, and output controls, forms the vocabulary you will use every day as an AI engineer. You will not always be tuning all of these at once. Most of the time you are adjusting temperature and occasionally Top-P. But the moment a model behaves unexpectedly, this knowledge is what tells you whether the problem is in how you are sampling, how much context you are feeding in, or how the output length is being constrained. Understanding these elements is what separates someone who uses AI tools from someone who can actually reason about and control them.