기술 가이드

자동 프롬프트 최적화 및 DSPy

Automatic prompt optimization uses an algorithm to search for better instructions or few-shot examples.

4분 읽기
마지막 업데이트 2026년 9월 15일

이 페이지에서4분 읽기

개요
심층 분석
전략적 영향
The Future of Automatic Prompt Optimization and DSPy
실제 구현
위험 및 가드레일
구현 로드맵
계속 탐색하세요
자주 묻는 질문

개요

It scores candidate prompts against a metric on a set of examples instead of relying on hand-tweaked wording. Well-known approaches include APE, Google DeepMind's OPRO and DSPy, an open-source framework from Stanford. Together they turn prompt engineering from guesswork into a measurable, repeatable process.

심층 분석

Hand-tuned prompts have a known weakness. Wording that works for one model, dataset or version can quietly stop working on another, and changes are usually judged on only a handful of examples. Automatic prompt optimization replaces that loop with a search. You provide three things: - A task. - A set of inputs, ideally with expected outputs. - A metric, such as exact match or a rubric scored by another model. The optimizer generates candidate prompts, runs them, scores them and keeps the winners. Early examples made the idea concrete. APE (Automatic Prompt Engineer, Zhou et al., 2022) had a model propose many instructions from input-output examples, then kept the highest-scoring ones. OPRO (Optimization by PROmpting, Yang et al., Google DeepMind, 2023) used an LLM as the optimizer. Its meta-prompt listed earlier instructions with their scores, and the model proposed new ones. OPRO found instructions such as "Take a deep breath and work on this problem step-by-step," which scored well for one particular model on grade-school math. Optimized wording can be unintuitive and specific to a model. DSPy goes further by changing how programs are written. It was developed at Stanford NLP by Omar Khattab and collaborators and introduced in 2023. You declare what each step does with a signature, such as "question, context -> answer," and combine modules like Predict or ChainOfThought. An optimizer (earlier called a teleprompter), such as BootstrapFewShot or MIPROv2, then "compiles" the program by choosing demonstrations and instructions for each module to maximize your metric. Three misconceptions are worth correcting: - These tools remove human judgment. They do not, because the metric and examples define "better." - Optimized prompts are safe to trust. They can overfit a small dataset. - Optimization is free. It costs real API calls, sometimes many.

기술적 통찰력

In DSPy, what gets optimized is each module's prompt, which has two tunable parts: the instruction text and the demonstrations. BootstrapFewShot runs your program on the training inputs and keeps the traces where the metric passes. It uses those traces' intermediate inputs and outputs as few-shot examples. This means you can get demonstrations for internal steps you never labeled. MIPROv2 also proposes candidate instructions. It then searches over combinations of instructions and demonstrations, using Bayesian optimization with a surrogate model to decide which combinations to try. In practice you need three things: a metric function that returns a score or a boolean, a training set, and a separate held-out set to check for overfitting. Budget matters, because each trial is a batch of model calls. A common pattern is: 1. Optimize with a cheaper model when possible. 2. Verify on the target model. 3. Save the compiled program. 4. Rerun the optimizer whenever the model or the data changes.

전략적 영향

비용 및 예산

아키텍처 결정은 수년 동안 성능과 운영 비용을 결정합니다.

더 명확한 결정들

기술 교육은 팀이 최신 스택뿐만 아니라 올바른 스택을 선택하는 데 도움이 됩니다.

품질 관리

더 나은 엔지니어링 선택은 생산 시 신뢰성 사고를 줄입니다.

The Future of Automatic Prompt Optimization and DSPy

Prompt optimization is becoming a normal part of LLM engineering, alongside evaluation suites, rather than a research curiosity. Open questions remain. How well do optimized prompts transfer between models? How do you write metrics for subjective qualities like tone or helpfulness? How do you avoid overfitting small datasets? Some research combines prompt optimization with lightweight fine-tuning, and DSPy supports both. Models change often, so the ability to re-optimize automatically against a stable test set is likely to matter more than any single clever prompt. Defining good metrics and examples remains a human job.

실제 구현

A team building a support-ticket classifier writes a DSPy signature "ticket -> category" and supplies 200 labeled tickets and an accuracy metric. An optimizer then picks the few-shot demonstrations that score best.

When a company moves from one LLM provider to another, it reruns its DSPy optimizer for the new model. It does not hand-rewrite prompts that were tuned for the old model.

A researcher runs an OPRO-style loop. One model proposes new wordings for an instruction, each wording is scored on a math benchmark, and the best-scoring history goes back into the next round of proposals.

A retrieval-augmented question-answering pipeline is optimized end to end, with a metric that checks final answers against reference answers. Both the query-writing step and the answering step improve together.

위험 및 가드레일

하나의 벤치마크를 최적화하면 더 광범위한 시스템 약점을 숨길 수 있습니다.
인프라 및 유지 관리 비용은 종종 과소평가됩니다.
시스템이 더욱 복잡해짐에 따라 보안 및 관찰 가능성의 격차가 커질 수 있습니다.

구현 로드맵

구현하기 전에 지연 시간, 품질, 비용 목표를 정의하세요.
현실적인 로드 및 데이터 조건에서 벤치마킹합니다.
오류, 드리프트 및 사용자 영향에 대한 계측기 모니터링.
확장하기 전에 롤백 및 사고 대응 경로를 준비하세요.

계속 탐색하세요

AI 벤치마크

기술 옵션을 비교할 때 평가를 올바르게 사용하십시오.가이드 읽기

강화 학습

기술 교육 전략에 대해 자세히 알아보세요.가이드 읽기

Free newsletter

Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.

One email each weekday. Unsubscribe in one click. We never sell or share your address.

Test yourself

Take the Automatic Prompt Optimization and DSPy quiz

Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.

퀴즈 시작

Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation

자주 묻는 질문

What is Automatic Prompt Optimization and DSPy?

Automatic prompt optimization uses an algorithm to search for better instructions or few-shot examples. It scores candidate prompts against a metric on a set of examples instead of relying on hand-tweaked wording. Well-known approaches include APE, Google DeepMind's OPRO and DSPy, an open-source framework from Stanford. Together they turn prompt engineering from guesswork into a measurable, repeatable process.

What must you supply for automatic prompt optimization to work?

The optimizer needs a task, examples and a metric so it can score candidate prompts and keep the best ones.

In OPRO, what plays the role of the optimizer?

OPRO uses a language model as the optimizer. It sees past instructions with their scores and proposes new ones.

What does OPRO's "Take a deep breath and work on this problem step-by-step" result illustrate?

The instruction scored well for one particular model on grade-school math. Wording found by search can be surprising and may not carry over to other models.

What is a DSPy signature?

Signatures declare what a module should do in terms of its input and output fields. DSPy then works out the prompt.

How does DSPy's BootstrapFewShot get its demonstrations?

Successful traces become few-shot examples, including for internal steps you never labeled yourself.

계속 학습하세요