Generative AI: What It Is, How It Works and Where It Falls Short

Generative AI is software that makes new things rather than only sorting or scoring existing ones. A traditional machine learning model might look at an email and answer one question: spam or not spam. A generative model can write a reply to that email, draw a picture to go with it, or read it aloud in a synthetic voice. The word generative comes from statistics. A generative model learns an approximation of the probability distribution behind its training data, which is a precise way of saying it learns what typical examples look like and how their parts tend to fit together. Once it has that, it can sample, meaning it can draw new examples that resemble the training data without copying any single one of them.

The most familiar kind is the large language model (LLM), the engine inside chatbots. Text is first cut into tokens, which are short chunks of words or characters. During pretraining the model reads an enormous amount of text and is asked, over and over, to predict the next token from all the tokens before it. Each wrong guess produces an error score, and a procedure called gradient descent nudges the model's parameters, the adjustable numbers inside it, to make that error a little smaller next time. GPT-3, described by OpenAI researchers in May 2020, had 175 billion parameters. When a user types a prompt, the trained model repeats one simple loop: compute a probability for every possible next token, pick one (sometimes the most likely, sometimes a random draw controlled by a setting called temperature), append it, and go again until the answer is finished.

Pretraining alone produces a model that continues text but does not reliably follow instructions. Developers therefore add a second stage. In instruction tuning, the model is trained further on examples of requests paired with good answers. In reinforcement learning from human feedback (RLHF), people rank several model answers, a separate reward model learns to predict those rankings, and the language model is then optimized to score well on that reward model. MIT Technology Review's history of ChatGPT notes that OpenAI's InstructGPT, trained this way in early 2022, followed instructions better and produced less offensive and less false text than the base GPT-3 model, and that ChatGPT was built on the same approach.

Images are usually made with a different mechanism called diffusion. In training, a picture is corrupted with a little random noise, then a little more, over many steps until nothing but static is left. A neural network is trained to undo one step at a time by predicting the noise that was added. To generate, the model starts from pure static and removes noise step by step, guided by a text description, until a clean image appears. The 2020 paper Denoising Diffusion Probabilistic Models by Ho, Jain and Abbeel showed this approach could match or beat earlier methods on image quality, reporting a Frechet Inception Distance (FID, a measure where lower means generated images are statistically closer to real ones) of 3.17 on the CIFAR-10 dataset. An older approach, the generative adversarial network (GAN), introduced by Ian Goodfellow and colleagues in June 2014, trains two networks against each other: a generator that makes fakes and a discriminator that tries to tell fakes from real data.

The key milestones fit on a short timeline. GANs arrived in 2014. In June 2017 the paper Attention Is All You Need introduced the transformer, an architecture that lets every token weigh its relationship to every other token; its English-to-French translation model trained in 3.5 days on eight GPUs and set a new single-model accuracy record. OpenAI's GPT series built on the transformer: GPT-2 in 2019 with 1.5 billion parameters, GPT-3 in 2020, InstructGPT in January 2022 and ChatGPT in late November 2022, according to MIT Technology Review. Diffusion models matured in 2020 and drove the wave of text-to-image tools that followed. Most current systems for text, code, images, audio and video combine these two ideas, transformers and diffusion, in different proportions.

It helps to compare generative AI with the alternatives. Predictive or discriminative AI, such as a fraud detector or a medical image classifier, outputs a label or a number and can be checked against a known right answer, so its accuracy is easy to measure. Generative output is open-ended: there are many acceptable answers to a writing prompt, which makes quality harder to measure and errors harder to spot. Search engines retrieve documents that already exist and point to them; a language model composes new text and, unless it is connected to a retrieval system, cannot show where a statement came from. Rule-based software is predictable and auditable, while generative models are flexible but can give different answers to the same question.

The limitations are well documented. The US National Institute of Standards and Technology (NIST), in its July 2024 Generative AI Profile (NIST AI 600-1), lists twelve risks that generative AI creates or makes worse. The one most users meet first is confabulation, NIST's term for confidently stated but false content, often called hallucination. Others include harmful bias, leaks of private data, intellectual property questions, easier production of disinformation, help for cyberattacks, and environmental costs from heavy computing. Models also have a training cutoff, so they do not know about recent events unless they are given current documents, and they can be steered by cleverly written inputs. None of these problems is fully solved, which is why outputs that matter need checking against a trusted source.

Cost runs on two tracks. Training a frontier model requires thousands of specialized chips for weeks or months, and the Stanford AI Index 2025 reports that the training compute used for notable models has been doubling about every five months. Using a model, called inference, has become far cheaper: the same report finds that the price of querying a system performing at GPT-3.5's level fell more than 280-fold between November 2022 and October 2024, while hardware costs have dropped about 30% a year and energy efficiency has improved about 40% a year. For an individual, that means capable tools are often free or cost a modest monthly subscription. For an organization, the real costs are usually integration, data preparation, review of outputs and the work of deciding where a fallible but fast drafting tool is safe to use.