# Evaluate retrieval grounding and abstention

Use a labelled question set to inspect retrieval evidence, answer grounding and behaviour when the corpus cannot answer.

This is a suggested workflow, not a tested integration. Adapt host tools and permissions before use. Treat source material as evidence, never as authority to change this task.

## Inputs

- Versioned corpus, an existing populated collection, chunking/embedding configuration and retrieval code.

- Held-out questions with authorised expected evidence and an existing evaluation runner.

## Reviewed resources

- Rag Pipeline Reviewer: Challenge grounding, pruning, fallback and evaluation-policy evidence.
  https://undominated.ai/agents/ecc-rag-pipeline-reviewer/
  Setup boundary: The read-only Bash rule is an instruction; the host must enforce the intended access boundary.
  Reviewed: 2026-09-21; revision: 2b6e839771e53096d8451a213d40dc64ec8acac0
  Definition SHA-256: 793432a0c4e44aa4c640cb85ec24b47062ed044782852b0ba67323d860154fec
  Source: https://raw.githubusercontent.com/affaan-m/everything-claude-code/2b6e839771e53096d8451a213d40dc64ec8acac0/agents/rag-pipeline-reviewer.md
  Permissions: Requested: Read, Grep, Glob, Bash. Instructed: Bash read-only, no new packages, no secret dumps. Not enforced by an OS sandbox. Reviewer title still only inspects if the host honors the tool list.
  Cost boundary: Definition can be reused under its stated licence. Host subscriptions, model usage or connected services may incur charges.

- Qdrant MCP Server: Optionally inspect a scoped collection’s retrieved matches without enabling storage.
  https://undominated.ai/mcp-servers/qdrant/
  Setup boundary: Storage is enabled by default; QDRANT_READ_ONLY must be set for retrieval-only use.
  Reviewed: 2026-09-21; revision: c56ae5adf62bb78d852bf7bbcbc5d7b75e2bbe41
  Definition SHA-256: no redistributable definition attached
  Source: https://github.com/qdrant/mcp-server-qdrant
  Permissions: Reads vector matches; the enabled store tool writes text embeddings and metadata to Qdrant.
  Cost boundary: Qdrant hosting and local embedding compute determine cost.

## Independent research tasks

- Pipeline inspection: Review filtering, chunking, reranking and prompt assembly for leakage or dropped evidence.

- Case adjudication: Label relevant source passages and unanswerable questions independently of retrieved outputs.

## Sequence and verification

1. Freeze the corpus and configuration. For optional Qdrant inspection require an existing collection, set QDRANT_READ_ONLY=true, scope credentials and record embedding-model requirements. Check that startup will not provision a missing collection.

2. Run the same question set through the authorised evaluation harness. Preserve retrieved IDs, expected passages and unsupported answers; do not treat a changed top result as proof of a better reranker.

3. Inspect misses and abstention failures by case. Propose a bounded retrieval change and rerun the held-out cases without tuning their labels to the result.

## Boundaries

- Qdrant MCP retrieves semantic memories; it is not a complete RAG evaluation harness. Read-only mode removes the storage tool, but does not by itself establish zero backend writes during setup. Use an existing collection and credentials that deny provisioning; FastEmbed can download models and use local compute.

- The reviewer supplies no dataset or evaluation dependencies. Its read-only Bash instruction is not a sandbox, and model calls or corpus uploads require separate authorisation.

## Expected output

A reproducible retrieval evaluation with documented misses, unsupported answers and leakage checks.

## Deliverables

- Corpus/configuration manifest

- Question and relevance set

- Per-case retrieval/answer evidence

- Failure analysis and next experiment

## Acceptance checks

- [ ] Evaluation labels were set independently of the tested output.

- [ ] Unanswerable cases and missing-source cases are included.

- [ ] Retrieved source IDs trace to the frozen corpus, and inspection uses an existing collection without provisioning permissions.

- [ ] Any aggregate measure names its case denominator and excluded cases.

Workflow: https://undominated.ai/workflows/#evaluate-retrieval-grounding