Evaluation engineering¶
All topics · Alphabetical index
Concepts¶
- Representative datasets; production examples; golden datasets
- Rubrics; golden answers; human annotation; inter-rater agreement
- Component evaluation; end-to-end evaluation; task completion rate
- Instruction adherence; retrieval relevance; groundedness; citation correctness
- Tool-selection accuracy; argument correctness; unauthorized-action rate
- Loop efficiency; unnecessary actions; recovery success
- Deterministic graders; model graders; judge calibration; judge bias
- Adversarial cases; slice analysis; regression suites
- Repeated runs; stochastic variation; confidence intervals
- Online experiments; shadow runs; canary evaluation
- Quality-latency-cost tradeoffs; evaluation coverage
Further reading¶
Reference links for this topic.