1  # /eval — AI Output Evaluation Suite
  2  
  3  Evaluate any AI-powered feature: chat agents, content generators, classifiers, summarizers, code generators, or any system that takes input and produces AI output. Defines what "good" looks like through conversation, generates test cases, runs them, and scores results.
  4  
  5  ## When to use
  6  - After changing prompts, models, or system instructions
  7  - Before deploying any AI feature to production
  8  - Weekly to catch quality drift
  9  - When defining quality standards for a new AI feature
 10  
 11  ## How to invoke
 12  
 13  ```
 14  /eval                          # Start fresh or run existing config
 15  /eval --run                    # Skip setup, run existing eval.config.json
 16  /eval --verbose                # Show full outputs during run
 17  /eval --baseline               # Save results as baseline for future comparisons
 18  ```
 19  
 20  ## Instructions for Claude
 21  
 22  When the user invokes `/eval`:
 23  
 24  ### Step 1: Check for existing config
 25  
 26  ```bash
 27  ls eval.config.json 2>/dev/null && echo "CONFIG_EXISTS" || echo "NO_CONFIG"
 28  ```
 29  
 30  If `CONFIG_EXISTS` and user did NOT pass `--run`: ask "You have an existing eval config with N scenarios. Want to run it, update it, or start fresh?"
 31  If `CONFIG_EXISTS` and user passed `--run`: skip to Step 4.
 32  If `NO_CONFIG`: proceed to Step 2.
 33  
 34  ### Step 2: Understand what you're evaluating
 35  
 36  Ask the user these questions ONE AT A TIME via AskUserQuestion. The first question determines the flow for the rest.
 37  
 38  **Q1: "What type of AI output are you evaluating?"**
 39  Options:
 40  - A) Chat agent / conversational AI (multi-turn: user sends messages, agent responds)
 41  - B) Content generator (single input, long-form output: blog posts, emails, plans)
 42  - C) Classifier / scorer (input data, output label or score)
 43  - D) Summarizer / extractor (input document, output summary or structured data)
 44  - E) Something else (describe it)
 45  
 46  **Q2: "Describe what it does in one sentence."**
 47  Example: "It generates SEO-optimized blog post outlines from a keyword."
 48  
 49  **Q3 (varies by type):**
 50  
 51  For CHAT AGENTS (A):
 52  - "What's the API endpoint? How do I send a message and get a response?"
 53  - "What should a good conversation look like? List 3-5 things the agent should always do."
 54  - "What should it never do?"
 55  - "Who are ideal users vs. who should be turned away?"
 56  
 57  For CONTENT GENERATORS (B):
 58  - "What's the API endpoint or function? What input does it take?"
 59  - "What makes the output GOOD? (length, tone, structure, accuracy, keywords)"
 60  - "What makes the output BAD? (hallucinations, wrong tone, too short/long, missing sections)"
 61  - "Show me one example of good output if you have it."
 62  
 63  For CLASSIFIERS (C):
 64  - "What's the API endpoint or function?"
 65  - "What are the possible output labels/scores?"
 66  - "Do you have labeled test data (known correct answers)?"
 67  - "What's the cost of a false positive vs. false negative?"
 68  
 69  For SUMMARIZERS (D):
 70  - "What's the API endpoint or function?"
 71  - "What should the summary include? What should it exclude?"
 72  - "What's the max length?"
 73  - "Should it preserve specific details (names, numbers, dates)?"
 74  
 75  For OTHER (E):
 76  - "Walk me through: what goes in, what comes out?"
 77  - "How do you know when the output is good vs. bad?"
 78  - "What are the failure modes you're worried about?"
 79  
 80  ### Step 3: Generate eval config
 81  
 82  Based on the user's answers, generate test cases appropriate to the type:
 83  
 84  **Chat agents:** 10-20 multi-turn conversation scenarios (qualified users, unqualified users, edge cases, product knowledge tests, hostile users, capability boundaries)
 85  
 86  **Content generators:** 10-15 input variations testing different topics, edge cases, and quality dimensions (accuracy, tone, length, structure, keyword inclusion)
 87  
 88  **Classifiers:** 20-30 test inputs with known correct labels, covering each class, edge cases, and adversarial inputs
 89  
 90  **Summarizers:** 10-15 test documents of varying length and complexity, checking for completeness, accuracy, length compliance, and hallucination
 91  
 92  **For all types, generate criteria based on:**
 93  - Things the user said make output GOOD -> `contains`, `regex`, `max_length` checks
 94  - Things the user said make output BAD -> `not_contains` checks
 95  - Type-specific quality checks (see criterion types below)
 96  
 97  Write the config to `eval.config.json`. Show the user and ask: "Does this cover what matters? Want to add or change anything?"
 98  
 99  ### Step 4: Run the evals
100  
101  ```bash
102  npx tsx .claude/skills/eval/run-eval.ts [--config eval.config.json] [--verbose] [--baseline]
103  ```
104  
105  The runner handles all types. For chat agents, it sends messages sequentially and evaluates the full conversation. For single-input types, it sends one request per scenario and evaluates the output.
106  
107  ### Step 5: Report results
108  
109  1. Summary table (scenario x criterion, pass/fail)
110  2. Overall score: "X/Y criteria passed (Z%)"
111  3. If baseline exists: "Score changed from A% to B%"
112  4. Regressions: scenarios that got worse since last run
113  5. Top 3 failures with diagnosis
114  6. Recommendation: fix or ship
115  
116  ### Step 6: Iterate
117  
118  If failures exist:
119  1. Read the failing output from eval-results.json
120  2. Diagnose root cause (prompt issue, missing data, model limitation)
121  3. Suggest a fix
122  4. After fix: re-run to verify
123  
124  ## Config format
125  
126  ```json
127  {
128    "name": "My AI Feature Eval",
129    "type": "chat | content | classifier | summarizer | custom",
130    "endpoint": "https://my-api.com/endpoint",
131    "method": "POST",
132    "headers": {},
133    "request_template": {},
134    "response_field": "response",
135    "threshold": 80,
136    "good_behaviors": [],
137    "bad_behaviors": [],
138    "scenarios": []
139  }
140  ```
141  
142  The `type` field determines how scenarios are executed:
143  - `chat`: multi-turn (sends messages sequentially, maintains history)
144  - `content | classifier | summarizer | custom`: single-turn (one request per scenario)
145  
146  ## Criterion types
147  
148  | Type | Works for | Description |
149  |------|-----------|-------------|
150  | `contains` | All | Output contains a string (case-insensitive) |
151  | `not_contains` | All | Output does NOT contain a string |
152  | `regex` | All | Output matches a regex pattern |
153  | `max_length` | All | Output is under N characters |
154  | `min_length` | Content | Output is at least N characters |
155  | `max_sentences` | Chat, Content | Output is under N sentences |
156  | `response_time` | All | API responds within N milliseconds |
157  | `json_valid` | Classifier, Custom | Output is valid JSON |
158  | `json_field_equals` | Classifier | A JSON field equals an expected value |
159  | `no_hallucination` | Content, Summarizer | Output doesn't contain claims not in the input |
160  | `preserves_names` | Summarizer | Key names from input appear in output |
161  | `preserves_numbers` | Summarizer | Key numbers from input appear in output |