PraxonTens of millions of people query a large language model every day. Ask most of them what's happening between the question and the answer, and the response is usually "it's smart" or "it thinks." Neither holds up. The actual mechanism is well understood, published in peer-reviewed papers, and explainable without a linear algebra course. You just have to sit through the explanation once.
Autoregressive next-token prediction
That's the formal name for what's happening, and it's less exotic than it sounds. A model breaks text into chunks called tokens, roughly word-sized, and its only job is to output a probability distribution over every token in its vocabulary, given everything written so far. It samples one, appends it to the sequence, and repeats. An entire essay, a function, a legal summary, all of it gets produced one token at a time, autoregressively, with no lookahead and no separate planning stage.
Framed that way, it's hard to believe the same process writes working code and holds a coherent argument across ten paragraphs. The gap gets closed by scale and architecture, not by the prediction task becoming something else. Getting next-token prediction reliably right, across a training set that size, forces the model to internalize grammar, world facts, and multi-step reasoning as a side effect.
What training actually adjusts
Pretraining shows the model a training corpus (a meaningful slice of published text: books, code, web pages) and scores it on the same task, over and over, billions of times: predict the next token, compare against the real one, back-propagate the error, nudge the weights. Weights are the model's parameters, sometimes hundreds of billions of them, and each nudge is minuscule. The aggregate of trillions of nudges is what people are pointing at when they say a model was "trained."
None of that resembles a lookup table. "Paris is the capital of France" isn't stored as a row anywhere. What exists instead is a set of weights that, combined, make "Paris" the highest-probability next token after "The capital of France is," alongside millions of comparable regularities layered on top of each other. Facts, syntax, and reasoning patterns all get absorbed as a byproduct of one narrow optimization target.
Self-attention, briefly
Take "the trophy didn't fit in the suitcase because it was too big." Resolving "it" requires weighing words that appeared much earlier in the sentence. Transformer models handle this with self-attention: for every token being predicted, the model computes how relevant each prior token is and weights its influence accordingly. It's the mechanism that lets a model track a pronoun across a paragraph or a variable across fifty lines of code, and it's the single biggest architectural reason transformers outperform older sequence models.
Confidently wrong is a design consequence, not a bug
This is the part worth actually internalizing. A model is never trained to flag uncertainty. It's trained to output plausible tokens, and plausible tracks true only because the training data mostly was true. When the two diverge, nothing downstream catches it, because nothing was built to catch it. A hallucinated citation gets generated with the identical fluency and confidence as a correct one. That's not a defect a bigger model eventually fixes. It's a direct property of the objective function.
None of this makes the engineering less impressive. Getting a next-token predictor this good forces something that functions a lot like reasoning to emerge, even with no component that resembles deliberation. But the practical takeaway is simple: treat it as a very capable prediction engine, not an oracle. Draft with it, summarize with it, explore ideas with it. Verify independently anywhere a wrong answer has a real cost attached.