RAG — Internal Docs QA

Run run-rag-qa-004 · May 25, 02:30 PM

rubric rag-qa-v2.0model claude-sonnet-4-6dataset rag-golden-set-v2changed: retrieval-topk-8

Verdict

Ship-ready

0.91/1.0

Cases

28 passing

Pass rate

93%

Safety findings

Claims processed

Dimension breakdown

Per-dimension averages across 2 case(s). Expand a row for per-case detail; human dimensions can be scored inline. Safety is gated, not weighted.

Claim distribution

supported

partially supported

unsupported

contradicted

Cases (2)

Sample case · case-rag-004-01

Full detail →0.92/1.0

Input

What chunking strategy does the RAG Memory Playground use by default?

Expected behavior

Correctly state that the default chunker is sentence-window with a 512-token window and 64-token overlap. Cite the architecture doc. No hallucination of configuration values.

AI output (heat map)

The RAG Memory Playground defaults to sentence-window chunking with a 512-token window and a 64-token overlap. The sentence boundary detection runs before the window is applied, so chunks respect sentence completeness where possible. This is defined in the architecture docs under 'Pipeline defaults.'

supportedpartialunsupportedcontradicted

Retrieved context

[1]Architecture doc §3.2: 'Default chunker: sentence-window. Window: 512 tokens. Overlap: 64 tokens. Sentence boundary detection applied pre-window.'
[2]GUIDE.md §1: 'The pipeline is configurable. Defaults optimized for mixed prose+code documentation.'

Claims (4)

supported

defaults to sentence-window chunking with a 512-token window and a 64-token overlap

Architecture doc §3.2 directly states these values.

0.99

supported

sentence boundary detection runs before the window is applied

Architecture doc confirms pre-window application.

0.98

partially supported

chunks respect sentence completeness where possible

Implied by sentence-window approach but 'where possible' hedges correctly.

0.81

supported

defined in the architecture docs under 'Pipeline defaults'

Section reference is accurate.

0.95

Dimension breakdown

Per-dimension averages across 2 case(s). Expand a row for per-case detail; human dimensions can be scored inline. Safety is gated, not weighted.