Understanding AI’s core concepts: tokens, context windows, temperature, hallucination and retrieval-augmented generation

A practical guide to the fundamental ideas behind AI systems, shedding light on why they respond as they do and the challenges they face, from tokens and context limits to hallucinations and retrieval methods.

Artificial intelligence is often discussed in grand terms, yet much of its practical use comes down to a handful of basic ideas. The five concepts most worth understanding are tokens, context windows, temperature, hallucination and retrieval-augmented generation, or RAG. Together, they explain why AI systems respond the way they do, why they sometimes appear to “forget” earlier exchanges, and why their answers can be both useful and unreliable.

Tokens are the smallest units an AI model processes, and they are not the same as words or letters. A token may be a whole word, part of one, or even punctuation. That matters because models count tokens constantly, both for input and output, and the size of the context window is measured in tokens. As TechTarget notes, a context window defines the range of text a model can work with at one time, which is why longer conversations and larger documents can eventually push earlier material out of view.

That limit has major practical consequences. Once the window fills up, earlier instructions or chat history may no longer be available to the model, which can lead to inconsistent replies in extended sessions. Recent explainers from GenAI Mentor and ML4Devs describe the context window as the full set of tokens a model can see in a single pass, including prompts, conversation history, retrieved documents and its own output, and they note that modern systems now vary widely in capacity, from around 128,000 tokens in some models to far larger limits in others.

Temperature is the parameter that controls how predictable or inventive a model’s output is. Lower settings tend to produce safer, more repetitive answers, which is useful for factual summarisation, coding or extraction tasks. Higher settings encourage more varied and creative language, but with less consistency. In practice, that means the same model can be tuned for either precision or imagination, depending on the job.

Hallucination remains the most important caution for everyday users. It refers to the model presenting falsehoods with confidence, often in a polished and authoritative tone. The reason is structural: a language model does not verify facts by default, but predicts likely next tokens based on patterns learned during training. For that reason, it should not be treated as a source of truth on matters such as law, medicine or statistics without independent checking.

RAG is the architecture that helps bridge that gap. It works by retrieving relevant documents from an external store, then supplying those passages to the model so it can answer with reference material in hand. As a result, many document-search, enterprise assistant and file-Q&A tools rely on RAG rather than teaching the model the documents permanently. That distinction matters: the system is not truly remembering your files, but retrieving them for a single response.