# Tokenizer Density Audit

Audit tokenizer vocabulary density, tokens-per-word / tokens-per-byte expansion ratios, and real effective text processing costs across different tokenizers (cl100k, o200k, Llama 3, Qwen, Claude BPE).

## Workflow

1. Measure raw corpus metrics: word count, character count, byte count.
2. Tokenize corpus with candidate models' tokenizers, recording generated token counts.
3. Compute density metrics:
   - Tokens per word: $D_w = \text{Tokens} / \text{Words}$
   - Bytes per token: $D_b = \text{Bytes} / \text{Tokens}$
4. Calculate effective text cost per million words:
   $$\text{Cost per Million Words} = D_w \times \text{Rate per Million Tokens}$$
5. Compare baseline vs candidate: reveal whether a nominally cheaper per-token model actually costs more to process the exact same text due to vocabulary inefficiency.

## Synthetic Input Format

JSON object with:
- `schemaVersion`: 1
- `benchmarkCorpus`:
  - `name`: string
  - `wordCount`: positive integer
  - `byteCount`: positive integer
- `evaluations`: list of objects, each containing:
  - `model`: string
  - `tokenizer`: string
  - `tokenCount`: positive integer
  - `ratePerMillionTokens`: non-negative decimal string
- `assertions`:
  - `maxTokensPerWord`: optional decimal string
  - `minBytesPerToken`: optional decimal string

## Exit Codes

- `0`: Validation passed and assertions satisfied.
- `1`: Assertion failed.
- `2`: Invalid input format or safety violation.

## Verification

```sh
python3 scripts/check.py examples/synthetic.json
```
