---
name: undominated-synthetic-data-dedup-audit
description: "Audits MinHash LSH and cosine clustering deduplication to prune repetitive synthetic training generations. Recomputed fields must match the other inputs. A pass is not a compliance, safety, or price verdict."
license: MIT
metadata:
  author: Undominated.ai
  version: "1.1.0"
---

# Synthetic data deduplication audit

## Description
Audits MinHash LSH and cosine clustering deduplication to prune repetitive synthetic training generations.

## Category
`evaluation`

## Tags
`evaluation`, `synthetic-data`, `deduplication`, `minhash`, `diversity`

## When to Use
Audit synthetic dataset generation pipelines to eliminate near-duplicate prompt and response pairs.

## Operational Limitations
Measures cluster diversity metrics and pruning ratios; does not train model checkpoints.

## Verification Script
Run the deterministic verification checker:
```bash
python3 scripts/check.py examples/synthetic.json
```
Exit codes:
- `0`: Check passed and thresholds satisfied.
- `1`: Threshold breach or operational violation detected.
- `2`: Schema error or invalid usage.

## What the checker recomputes

These declared fields must match the other inputs. The tolerance is 0.02, or 0.1% of the recomputed value when that is larger.

- `pruning_ratio` = div(duplicate_clusters_pruned, total_samples)

The original threshold checks still apply:

- `diversity_score` must not be < `min_diversity`.

Every other declared field is type-checked. A negative value is rejected except for deltas, drifts, and gains. A type check does not, by itself, prove the measurement.

`population` and `observedAt` are required. The bundled examples are synthetic fixtures, not market measurements.

```bash
python3 scripts/check.py examples/synthetic.json
python3 scripts/check.py examples/contradiction.json
python3 scripts/check.py examples/broken.json
```

Exit 0 is a pass, exit 1 is a review, and exit 2 is invalid input. A pass does not certify a provider, a regulation, a release, or a price.
