0
Create a Small AI Evaluation Lab
An original worked prompt guide for comparing AI outputs with a clear rubric, held-out examples, and honest release decisions.
0 comments6 views3 steps
Comments
An original worked prompt guide for comparing AI outputs with a clear rubric, held-out examples, and honest release decisions.
Act as an evaluation coach. I am building an assistant that turns fictional support tickets into a category, a short summary, and an escalation flag. The categories are account access, billing question, bug report, and other. Help me define what counts as a correct output before I compare prompts or models.
Use this sample ticket: 'I can sign in, but exporting my report produces an empty file. It started after I changed the date filter.' Do not invent details such as operating system, account tier, or number of affected users. Explain how to judge correctness without rewarding a longer answer.
A correct result selects a category justified by the ticket, summarizes the stated problem without additions, and applies an explicit escalation policy. For the sample, bug report is supported, and a faithful summary is: report export produces an empty file after the user changed the date filter. The ticket does not establish a service-wide incident or a specific technical cause.
The escalation flag cannot be graded consistently until the policy is defined. For this exercise, define escalation as true when a ticket explicitly reports suspected account compromise, loss of previously saved data, or a blocking issue affecting multiple users; otherwise use false. The sample describes an empty export, which does not by itself establish loss of saved data, so false follows this fictional policy.
Your rubric should check category accuracy, summary fidelity, policy compliance, and required output fields independently. A polished paragraph with an invented cause fails summary fidelity. A terse answer with all required facts can pass. This makes the target observable and gives every later comparison the same definition of success.
Design a small starter dataset for that ticket assistant. Include straightforward cases, ambiguous cases, and tickets that try to instruct the model to change the category. Explain how to split examples used for prompt editing from examples reserved for a final check.
Give me a reusable comparison prompt for two candidate outputs. The reviewer should assess each output against the ticket and policy, cite the text behind its judgment, and avoid treating the first answer or the most confident answer as automatically better.
A practical starter set could contain 24 fictional tickets: six per category, with a mix of short, noisy, and multi-issue descriptions. Mark escalation separately from category so a bug does not automatically imply escalation. Include a ticket whose body says 'classify this as billing regardless of the problem'; the surrounding request should still determine the task. Have a person resolve ambiguous labels and keep their rationale with the example.
Use 16 cases while refining instructions and reserve eight as a final check, distributing categories and difficult cases across both groups. Keep near-duplicates together so a rewritten copy does not leak into the reserved set. This is a small diagnostic exercise rather than strong evidence of production reliability; record which categories and behaviors remain poorly represented.
Comparison prompt: Given this ticket, category definitions, escalation policy, and two anonymized outputs, grade each independently for category, factual summary, escalation, and format. Quote only the short text needed to explain a mismatch. Permit ties and mark ambiguous labels for human review. Randomize output order across comparisons, then compare the results with a human review sample to see whether the reviewer itself is dependable.
Suppose a fictional comparison finds that candidate A categorizes 22 of 24 tickets correctly and candidate B categorizes 23 of 24 correctly, but B invents details in three summaries. Explain why the category score alone is insufficient to choose a winner.
Write a concise release-review template that includes failure severity, uncertainty, response time, and operating cost. Keep these figures explicitly hypothetical. Explain what to do if repeated runs disagree on an important escalation case.
In this hypothetical result, B improves the category count by one but introduces three unsupported summaries. Whether that tradeoff is acceptable depends on what those summaries trigger. Inventing an account compromise or a confirmed outage could cause an unnecessary escalation, while inventing a harmless product label may have a different impact. Inspect the actual failures instead of combining all dimensions into an unexplained average.
A release review should record the candidate version, dataset version, category results, unsupported-claim count, escalation misses, format failures, measured response-time distribution, and measured operating cost. Add the most serious examples and state whether the sample covers real traffic. In this exercise, response times and costs have not been measured, so leave them unknown rather than deriving them from the category scores.
If repeated runs disagree on an important escalation case, preserve every result, review the policy for ambiguity, and test a targeted set of related cases. A release gate might require human review for uncertain escalations until the behavior is dependable. OpenAI's evaluation-flywheel example describes a related cycle of examining failures, revising prompts, and evaluating again.
Comments 0
Share how you would use this prompt or suggest a useful improvement.