Automatic Prompt Optimization and DSPy Quiz

Answers and explanations appear as you take the quiz.

1. What must you supply for automatic prompt optimization to work?
   • Only a longer prompt
   • A task, a set of example inputs and a metric to score outputs
   • A fine-tuned model and a GPU cluster
   • A list of banned words
2. In OPRO, what plays the role of the optimizer?
   • A gradient descent step on the model's weights
   • A human prompt engineer
   • A random search with no feedback
   • An LLM given a meta-prompt listing earlier instructions and their scores
3. What does OPRO's "Take a deep breath and work on this problem step-by-step" result illustrate?
   • Optimized wording can be unintuitive and specific to a model
   • Breathing instructions work for every model
   • Math tasks do not need prompts
   • OPRO only works for writing tasks
4. What is a DSPy signature?
   • A cryptographic key for the API
   • A model's version number
   • A declaration of a step's inputs and outputs, such as "question, context -> answer"
   • A watermark on generated text
5. How does DSPy's BootstrapFewShot get its demonstrations?
   • It downloads examples from the internet
   • It runs the program on training inputs and keeps the traces where the metric passes
   • It asks the user to label every intermediate step
   • It uses the longest available examples
6. Why should you keep a separate held-out set when optimizing prompts?
   • To reduce the size of the prompt
   • To make the optimizer run faster
   • It is required by the API
   • To check whether the optimized prompt has overfit the training examples
7. What were DSPy's optimizers called earlier?
   • Teleprompters
   • Compilers
   • Tokenizers
   • Retrievers
8. What approach did APE (Automatic Prompt Engineer) take?
   • It fine-tuned model weights on instructions
   • It hand-wrote prompts from expert interviews
   • It had a model propose instructions from input-output examples and kept the highest-scoring ones
   • It removed instructions entirely