1. |
|
Which practices should engineers follow when managing prompts in a production AI system? (Select all that apply)
|
|
|
- Store prompts in version control (e.g., git) alongside code
- Run eval regression tests before promoting prompt changes to production
- Make ad-hoc prompt edits in production without logging changes
- Maintain separate prompts for dev, staging, and production environments
|
 Answer & Explanation
Answer: Option A & B & D
Explanation:
Production prompt management requires: version control (store prompts in git like code), eval regression testing before deploying changes, environment separation (dev/staging/prod prompts), and logging (record prompts + outputs for debugging). Ad-hoc manual updates without tracking cause silent regressions.
|
 View Answer |
|
2. |
|
In a multimodal LLM prompt, what additional input type can be provided beyond text?
|
|
|
- Images, audio, or video alongside text
- Only additional system prompts
- Raw SQL queries only
- Compiled binary executables
|
Answer & Explanation
Answer: Option A
Explanation:
Multimodal LLMs (e.g., GPT-4o, Claude 3, Gemini 1.5) accept image, audio, and/or video inputs alongside text. This enables tasks like visual QA, chart reading, image description, and document understanding without separate vision models.
|
| View Answer |
|
3. |
|
Why is controlling the number of output tokens important when deploying LLMs in production?
|
|
|
- More tokens always produce higher-quality answers
- Token limits only apply to embedding models, not chat models
- Output tokens drive API cost and inference latency; constraining them is essential for production efficiency
- Models automatically stop at an optimal length with no configuration
|
Answer & Explanation
Answer: Option C
Explanation:
Output tokens directly drive API costs and latency. Unconstrained generation can produce unnecessarily long responses, wasting money and slowing user experience. Setting max_tokens and instructing the model to be concise ensures cost-effective, fast production systems.
|
| View Answer |
|
4. |
|
A prompt instructs: 'Respond in under 100 words, use bullet points, target a 10-year-old audience.' What prompt design principle does this illustrate?
|
|
|
- Output constraint specification
- Few-shot prompting
- Chain-of-thought prompting
- Retrieval-Augmented Generation
|
Answer & Explanation
Answer: Option A
Explanation:
Adding explicit output constraints (length, format, audience level) is a core prompt engineering principle. These constraints narrow the model's output space, increase predictability, and align the response with downstream requirements without any code changes.
|
| View Answer |
|
5. |
|
What is the purpose of assigning a role such as 'Act as a senior security auditor' in a prompt?
|
|
|
- It shapes the model's knowledge register and communication style to match the assigned role
- It grants the model access to internet resources
- It encrypts the response for security purposes
- It reduces the token usage of the response
|
Answer & Explanation
Answer: Option A
Explanation:
Persona prompting (role assignment) shifts the model toward a specific register of knowledge and communication style. It leverages the model's representation of domain experts, leading to more focused, domain-appropriate responses without any additional training.
|
| View Answer |
|
6. |
|
Why should engineers A/B test prompt variants rather than choosing based on a single example?
|
|
|
- A single example may not represent overall performance due to model variance and distribution shift
- A/B testing always reduces API costs by 50%
- A/B testing is required by all LLM API providers
- A single example is sufficient if it looks good
|
Answer & Explanation
Answer: Option A
Explanation:
A single cherry-picked example can be misleading due to LLM non-determinism and distribution variation. A/B testing over a statistically representative eval set reveals which prompt variant performs better in aggregate and avoids overfitting to one case.
|
| View Answer |
|
7. |
|
What is the 'LLM-as-Judge' evaluation pattern?
|
|
|
- Using a (stronger) LLM to score target model outputs against a defined rubric
- Running keyword-match checks against expected output strings
- Using the same model to generate and evaluate its own responses in the same call
- Asking users to rate responses on a 1–5 scale inside the chat UI
|
Answer & Explanation
Answer: Option A
Explanation:
LLM-as-Judge uses a (typically stronger) LLM to score or rank the outputs of a target model against a rubric, replacing or augmenting expensive human evaluation. It is scalable but can inherit the judge model's own biases, so calibration against human labels is important.
|
| View Answer |
|
8. |
|
Which metrics are commonly used when evaluating LLM outputs? (Select all that apply)
|
|
|
- Factual accuracy against ground-truth answers
- Relevance to the input query
- Output format compliance with the specified schema
- Average typing speed of the prompt author
|
Answer & Explanation
Answer: Option A & B & C
Explanation:
Key evaluation dimensions include: factual accuracy (does the answer match ground truth?), relevance (does it address the question?), format compliance (is the output in the requested structure?), and consistency (does the model give the same answer to paraphrased questions?). Keyboard speed is irrelevant.
|
| View Answer |
|
9. |
|
What is the goal of prompt evaluation (evals) in an LLM project?
|
|
|
- To reduce the number of API calls made per user session
- To measure and benchmark prompt effectiveness using representative test cases
- To automatically fine-tune the model based on user feedback
- To encrypt prompt content before sending to the API
|
Answer & Explanation
Answer: Option B
Explanation:
Prompt evaluation systematically measures how well a prompt (and model) performs on a representative test set, using metrics like accuracy, relevance, factuality, and format compliance. Evals enable data-driven iteration and catch regressions when prompts or models change.
|
| View Answer |
|
10. |
|
What does the 'top_p' (nucleus sampling) parameter control in LLM generation?
|
|
|
- The maximum number of output tokens
- The number of fine-tuning epochs
- The number of parallel inference threads
- The cumulative probability threshold for the token candidate pool
|
Answer & Explanation
Answer: Option D
Explanation:
Top-p (nucleus) sampling restricts token selection to the smallest set of tokens whose cumulative probability exceeds p. For example, top_p=0.9 means only tokens in the top 90% probability mass are considered. It produces more coherent diversity than pure top-k sampling.
|
| View Answer |
|
