How to Analyze Survey Results with AI

Survey data comes in two kinds, and they need different handling. Closed questions, such as multiple choice or 1-to-5 agreement scales, are counting problems. Their totals should come from a spreadsheet or from code the AI runs, never from the model estimating. Open-ended answers are a reading problem, and this is where language models help most. Researchers have long analyzed open-ended answers through a process called coding. They build a codebook, a list of named themes with definitions, and label each response with one or more themes. Deductive coding starts from categories you expect. Inductive coding lets themes emerge from what people actually wrote. A language model can do a fast first round of inductive coding. A reliable workflow: have the model read a sample and propose themes, review and edit the codebook yourself, have the model apply that fixed codebook to every response, and check a sample of its labels by hand. The common failures are predictable. A model may write that 'about 40% mentioned price' without counting anything. It may produce quotes that sound real but are rewordings or blends of several responses. Given more text than it can handle at once, it may silently skip responses. It may also treat the average of a 1-to-5 scale as the full story, even though the spread of answers often matters more. Many people assume that if the AI states a number, it counted. Unless it ran code over labeled data you can inspect, it probably did not. AI also cannot fix problems with who answered. If only your happiest customers responded, a perfect analysis still describes only them. Be cautious with subgroups too: a percentage based on eight people can swing a lot when one person answers differently.