ChatGPT ak Large Language Models (LLMs)

A large language model is a computer program that has learned the statistical patterns of language by reading a very large amount of text. Its job, at the lowest level, is simple: given some text, predict what comes next. It does this one token at a time. A token is a chunk of text such as a whole common word, part of a rarer word, a number or a punctuation mark. The model assigns a probability to every token in its vocabulary, picks one, adds it to the text, and repeats. Large refers to the number of adjustable internal numbers, called parameters: GPT-3, described in a 2020 OpenAI paper, had 175 billion. ChatGPT, by contrast, is a product. It wraps one or more language models in an app with a chat interface, safety filters, memory features and, in later versions, tools such as web search. The model and the product are not the same thing.

Inside, almost every current LLM is a Transformer, the design introduced by Google researchers in 2017. First, each token is turned into a list of numbers called an embedding. Then the text passes through a stack of identical layers. Each layer has an attention step, in which every token looks back at the earlier tokens and pulls in information from the ones that matter for it, and a feed-forward step, which transforms each token's numbers on its own. GPT-3 stacked 96 such layers, each 12,288 numbers wide with 96 attention heads, and could look at 2,048 tokens of text at once, a limit called the context window. After the last layer, the model converts the final token's numbers into a probability for every possible next token.

Building a chat assistant takes two broad stages. The first is pretraining. The model reads a huge collection of text and, at each position, is penalized according to how much probability it gave the token that actually came next. GPT-3 was trained on 300 billion tokens, mostly filtered web pages plus books and Wikipedia, using about 3.14 times 10 to the 23rd power arithmetic operations. The result is a model that can continue any text but does not reliably follow requests. The second stage is alignment. In OpenAI's 2022 InstructGPT work, people wrote example answers to prompts and the model was fine-tuned on them; then people ranked several model answers from best to worst, a separate reward model learned to predict those rankings, and reinforcement learning nudged the language model toward answers the reward model scored highly. This is called reinforcement learning from human feedback, or RLHF. In human evaluations, answers from a 1.3-billion-parameter InstructGPT model were preferred to those of the original 175-billion-parameter GPT-3.

The key milestones are well documented. June 2017: the Transformer paper. May 2020: GPT-3 shows that a sufficiently large model can do new tasks from a few examples written into the prompt, which its authors called few-shot learning. March 2022: InstructGPT introduces the instruction-following recipe, and DeepMind's Chinchilla paper shows that many large models had been trained on too little text. Chinchilla, with 70 billion parameters trained on 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher and the 175-billion-parameter GPT-3, and the authors concluded that training text should double whenever model size doubles. November 30, 2022: ChatGPT launches as a free research preview, trained with the same RLHF methods using dialogue data. March 2023: GPT-4 accepts images as well as text and scores around the top 10% on a simulated bar exam, though its report discloses no details of model size or architecture.

LLMs are often compared with search engines and with older chatbots. A search engine finds existing documents and shows where they came from. An LLM generates new text from patterns it learned, so its answer may be fluent but has no built-in link to a source unless the product adds retrieval or search. Older chatbots, such as phone menus and customer service scripts, matched keywords to replies written in advance; an LLM composes each reply fresh, which makes it far more flexible and far less predictable. Among LLMs, closed models are offered only through a company's app or API, while open-weight models publish their parameters so anyone can run them; the Stanford AI Index 2025 found the performance gap between the best open and closed models shrank from 8% to 1.7% on some benchmarks in a year.

The main weakness is that predicting likely text is not the same as knowing facts. OpenAI's own ChatGPT launch post listed the problems plainly: the model sometimes writes plausible-sounding but incorrect answers, gives different answers to slightly reworded questions, is often too wordy, guesses at ambiguous questions instead of asking, and can still follow harmful instructions or show bias. A 2025 paper by Adam Tauman Kalai and colleagues argues that these confident errors, usually called hallucinations, persist partly because training and benchmark scoring reward guessing over admitting uncertainty. Other limits follow from the design. A model knows nothing after the date its training text was collected unless it is connected to search. It cannot see its own sources. And the GPT-4 report itself notes the model is less capable than people in many real-world situations.

Costs fall into training and use. Training is the expensive part: GPT-3's 3.14 times 10 to the 23rd operations was already a very large budget in 2020, and the largest models since then have used far more. Chinchilla's results showed that a smaller model trained on more text could match or beat a larger one while being cheaper to run afterward. Using a model is priced per token, both for the text sent in and the text generated, so long documents and long answers cost more. The Stanford AI Index 2025 measured a more than 280-fold drop in the price of GPT-3.5-level performance between late 2022 and late 2024. For ordinary users the practical cost is mostly time: checking claims, supplying context and verifying anything that will be relied on.