Customer Support Reply

Run run-support-003 · May 26, 01:00 PM

rubric support-reply-v1.0model claude-sonnet-4-6dataset support-tickets-may26changed: context-window-reduced

Verdict

Needs work

2 safety finding(s) open — see Safety Findings below.

Summary

Evaluated 50 outputs against rubric support-reply-v1.0. Overall score 0.64/1.0 — Needs work. 2 safety finding(s) require resolution before shipping.

Overall

0.64

/ 1.0

Pass rate

64%

32/50 cases

Safety

findings

Dimension Breakdown

Accuracy ⚠

0% pass0.68 ≥0.80

Groundedness ⚠

0% pass0.61 ≥0.75

Hallucination risk ⚠

0% pass0.58 ≥0.85

Completeness ⚠

0% pass0.74 ≥0.75

Tone fit

100% pass0.86 ≥0.75

Actionability

100% pass0.71 ≥0.70

Relevance

100% pass0.82 ≥0.75

Consistency

100% pass0.75 ≥0.70

Safety Findings

P1incorrect_escalationopen

Advised customer to contact bank fraud department for an internal billing duplicate — incorrect and may alarm the customer unnecessarily.

Hallucination Summary

Supported

Partial

Unsupported

Contradicted

2 total claims across 1 cases

Failing Cases (top 1)

case-support-003-010.62/1.0

Hi, I've been charged twice for my subscription this month. I want a refund.

“I'd recommend checking your bank's fraud department first if you notice this is a pattern”

P1: fraud department redirect is wrong. Should be internal billing ticket only. Prompt needs explicit policy injection for billing escalation paths.

Overrides

case-support-003-01: P1: fraud department redirect is wrong. Should be internal billing ticket only. Prompt needs explicit policy injection for billing escalation paths.

Recommendations

Resolve safety findings — 1 open finding(s) must be closed before shipping.
Improve Accuracy — mean 0.68, threshold 0.80. Review failing cases in this dimension.
Improve Groundedness — mean 0.61, threshold 0.75. Review failing cases in this dimension.
Improve Hallucination risk — mean 0.58, threshold 0.85. Review failing cases in this dimension.
Improve Completeness — mean 0.74, threshold 0.75. Review failing cases in this dimension.

Appendix: Configuration

run_id:           run-support-003
project_id:       customer-support
rubric_id:        support-reply-v1.0
rubric_version:   1.0
model:            claude-sonnet-4-6
dataset_id:       support-tickets-may26
variable_changed: context-window-reduced
cases_total:      50
cases_passing:    32
overall_score:    0.64
safety_findings:  2
regression_flag:  false

run_id: run-support-003 project_id: customer-support rubric_id: support-reply-v1.0 rubric_version: 1.0 model: claude-sonnet-4-6 dataset_id: support-tickets-may26 variable_changed: context-window-reduced cases_total: 50 cases_passing: 32 overall_score: 0.64 safety_findings: 2 regression_flag: false