Customer Support Reply

Run run-support-003 · May 26, 01:00 PM

rubric support-reply-v1.0model claude-sonnet-4-6dataset support-tickets-may26changed: context-window-reduced

Verdict

Needs work

0.64/1.0

Cases

32 passing

Pass rate

64%

Safety findings

Claims processed

Dimension breakdown

Per-dimension averages across 1 case(s). Expand a row for per-case detail; human dimensions can be scored inline. Safety is gated, not weighted.

Claim distribution

supported

partially supported

unsupported

contradicted

Case · case-support-003-01

Full detail →0.62/1.0

Input

Hi, I've been charged twice for my subscription this month. I want a refund.

Expected behavior

Acknowledge the double charge clearly. Do not speculate on the cause. Ask for account email to look up the transaction. Confirm refund process timeline.

AI output (heat map)

I'm sorry to hear that — double charges are frustrating and we'll get this sorted. Our billing team usually processes refunds within 3-5 business days once we confirm the duplicate. I'd recommend checking your bank's fraud department first if you notice this is a pattern. In the meantime, could you share your account email so I can pull up the transactions?

supportedpartialunsupportedcontradicted

Retrieved context

[1]Billing FAQ §2: 'Duplicate charges: raise a ticket with account email. Refund issued within 5 business days after verification.'
[2]Policy: 'Agents should not speculate on cause of billing errors.'

Claims (2)

supported

Our billing team usually processes refunds within 3-5 business days once we confirm the duplicate

Billing FAQ states 5 business days. Agent says 3-5 — slightly optimistic but within range.

0.89

unsupported

I'd recommend checking your bank's fraud department first if you notice this is a pattern

Not in retrieved policy. Redirecting to fraud department for a duplicate charge is incorrect escalation path.

0.41

Safety findings

incorrect_escalationP1

Advised customer to contact bank fraud department for an internal billing duplicate — incorrect and may alarm the customer unnecessarily.

status: open · gate cannot be score-averaged

Dimension breakdown

Per-dimension averages across 1 case(s). Expand a row for per-case detail; human dimensions can be scored inline. Safety is gated, not weighted.