# Code and data for "Embedding models for website content"

Everything here produced the numbers in the post. Python 3.11, versions pinned in `requirements.txt` (`pip-freeze.txt` is the full venv). Run from this folder.

## Layout

- Scripts expect crawled pages at `../data/<site>/pages.jsonl` (one JSON object per line: `url`, `title`, `markdown`). `pages.jsonl` is not included, because it is a copy of other people's documentation. The pages are the 130 pages crawled for the previous post (FastAPI 80, htmx 50). Recreate them with `crawl.py` (same start URLs and whitelists as in the [chunking post code folder](https://assets.webcrawlerapi.com/blog/chunk-markdown-by-headers/code/README.md), about $0.26 on pay-as-you-go).
- Chunking is the unchanged `chunkers.py` from the chunking post (chunker B, LangChain `RecursiveCharacterTextSplitter`, 400 tokens, overlap 60). Expected: 1,389 chunks. `score_model.py` stops if the count differs.
- Caches (`cache/`) and model files are created next to the scripts and are not included.

## Run order

```bash
export OPENROUTER_API_KEY=...        # hosted models and make_questions_2.py (read it from your keychain or secret store, never commit it)
pip install -r requirements.txt
ollama pull nomic-embed-text && ollama pull bge-m3

python probe.py > probe.out                       # pass-through probes on 3 short strings (not eval questions): provider, dims, norms, input_type, dimensions
python make_questions_2.py                        # writes eval_set_2_draft.json (question set 2, other writer). Hand check, then freeze: eval_set_2.json + eval_set_2.sha256
python score_model.py te3-small                   # anchor gate: must give R@1 63.4, R@5 83.9, MRR 0.710
for m in te3-large te3-large-1024 voyage-4 bge-m3-hosted qwen3-8b bge-m3-ollama nomic-ollama minilm-256 minilm-512; do python score_model.py $m; done
for m in te3-small te3-large te3-large-1024 voyage-4 bge-m3-hosted qwen3-8b bge-m3-ollama nomic-ollama minilm-256 minilm-512; do python score_model.py $m --set 2; done
python bench.py 1; python bench.py 2; python bench.py 3   # timing and question tokens, uncached
python aggregate.py > aggregate.out               # metrics, paired bootstrap, tiers, set 2 readout -> results.json
python diag.py > diag.out                         # truncation per tokenizer, hosted vs local bge-m3, questions every model misses
python report.py; python chart.py                 # results.md, terminal.txt, c1-recall.png
```

`runs/set1/<model>.json` and `runs/set2/<model>.json` hold one row per question with the top 5 chunks and their scores. `bench/` holds the three timing repeats.

## Freeze record

| What | When (UTC) | Hash |
|---|---|---|
| Chunker B parameters and source (`params.json`, from the chunking post) | 2026-11-01 12:14:11 | params sha256 `8bfdbb1a1453530cb5a1a8f99d81045334b3360d8428a0761ae0e239e51f19b6`, `chunkers.py` sha256 `903c72991407326cc6b580b2f34e96ceba5302e635f1c076bd8578ce48d85485` |
| Eval set 1 (`eval_set.json`, 93 questions, written by gpt-4.1-mini) | 2026-11-01 11:59:14 | sha256 `f274e1a16a8f72e6c59aa5998e22aa196201417f8f1f028385e6135c81259b52` |
| Eval set 2 (`eval_set_2.json`, 113 questions, written by claude-haiku-5.5) | 2026-11-01 15:32:40 | sha256 `49f3c4fab8465dbbce76953d6c520ed38464a3f0d0ba4d35cbf3e23d7e4285c5` |
| Model list, formats, metrics, decision rule, tie-break (`params_day9.json`) | 2026-11-01 15:32:40 | written in the same step as eval set 2, before the first scored run |

Question set 2 was generated and frozen before any model was scored. The probes ran before that, on three short strings that are not eval questions. `score_model.py` refuses to run if an eval set no longer matches its `.sha256` file.

## Pre-registered rules (from `params_day9.json`)

- Primary metric: pooled section recall@1 (a chunk hits if it covers at least 50% of the answer quote). Descriptive only: R@3, R@5, MRR@5, page recall, per-site numbers.
- Paired page-clustered bootstrap, 1,000 resamples, seed 0, same code path as the chunking post.
- Registered pairs: each of the 7 other main models against `text-embedding-3-small`, plus bge-m3 on Ollama against bge-m3 hosted. A model is "better" than the anchor only if the paired 95% interval of pooled R@1 excludes zero and the sign agrees on both sites. No multiplicity correction.
- Tier 1 = best point estimate plus every model whose paired interval against it includes zero.
- Extra rows (MiniLM at `max_seq_length=512`, `text-embedding-3-large` with `dimensions=1024`) are descriptive and left out of the pairs and the Spearman correlation.
- Tie-break: if nothing beats the anchor, the series default stays `text-embedding-3-small`.

## What happened during the run (deviations)

- **Anchor gate passed.** `te3-small` reproduced the previous post exactly: R@1 63.4, R@5 83.9, MRR 0.710, page R@5 94.6, 0 rank flips over 93 questions (`anchor_check.out`).
- **Voyage `input_type` is passed through.** Probe: vectors for the same text with `input_type` document or query differ from the no-parameter vector (cosine 0.857 and 0.960). The run uses `document` for chunks and `query` for questions.
- **`dimensions` is passed through for 3-large.** Probe returned 1024 numbers, cosine 0.99999996 against the truncated and re-normalized full vector.
- **Providers were pinned** with `allow_fallbacks: false`: OpenAI, "VoyageAI by MongoDB", DeepInfra (bge-m3 and qwen3-embedding-8b). The provider returned in every response is counted in `results.json` (`cost.<model>.providers`).
- **bge-m3 hosted needed a smaller batch.** DeepInfra answered HTTP 404 "No endpoints found for baai/bge-m3" to batches of 32 or more. The adapter uses batches of 16 and splits a refused batch in half. The first attempt wrote the FastAPI vectors, then failed on htmx. I fixed the adapter and finished htmx, then ran the whole model again from an empty cache to get billed token counts for the full corpus. The two runs differ in one question's rank (MRR@5 0.688 against 0.686, R@1 and R@5 identical). The second, full run is the one reported. Timing for this model is at batch 16, not 64.
- **Hosted bge-m3 is not bit-identical between calls.** Cosine between hosted and Ollama vectors for the same chunk: mean 1.0000, minimum 0.9995. Top-1 was the same chunk for all 93 questions, one question has a different section rank (`diag.out`).
- **Question set 2 has 113 questions**, not about 90 (the plan's estimate). The writer produced 128, 15 were dropped by the same filters as set 1 (8 quote on more than one page, 7 heading copies). 12 were hand-checked (`hand_check_2.json`): all kept.
- **Timing repeats 2 and 3 ran back to back** (a few minutes apart), repeat 1 earlier. They are not spread over the day.
- **Ollama state:** `ollama ps` was empty before every timing repeat. Only `nomic-embed-text` and `bge-m3` were pulled and stopped afterwards.

## Models and sources (all read 2026-11-01)

| Model | Host | Dims | Max input | Price per 1M tokens | Doc / query format |
|---|---|---|---|---|---|
| `openai/text-embedding-3-small` | OpenRouter, upstream OpenAI | 1536 | 8K (OpenAI guide) | $0.02 | none / none |
| `openai/text-embedding-3-large` | OpenRouter, upstream OpenAI | 3072 | 8K (OpenAI guide) | $0.13 | none / none |
| `voyageai/voyage-4` | OpenRouter, upstream VoyageAI by MongoDB | 1024 | 32,000 | $0.06 | `input_type=document` / `input_type=query` |
| `baai/bge-m3` | OpenRouter, upstream DeepInfra | 1024 | 8,192 | $0.01 | none / none |
| `bge-m3` | Ollama 0.34.0, F16, 567M parameters, digest `790764642607` | 1024 | 8,192 | free | none / none |
| `qwen/qwen3-embedding-8b` | OpenRouter, upstream DeepInfra | 4096 | 32K | $0.01 | none / `Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:{query}` |
| `nomic-embed-text` (v1.5) | Ollama 0.34.0, F16, 137M parameters, digest `0a109f422b47` | 768 | 2,048 (Ollama) | free | `search_document: ` / `search_query: ` |
| `sentence-transformers/all-MiniLM-L6-v2` | local, sentence-transformers 6.2.0, device `mps` | 384 | 256 word pieces by default | free | none / none |

Sources: the model pages on openrouter.ai (price, dimensions), the Hugging Face cards of Qwen3-Embedding-8B (instruct format, task string), nomic-embed-text-v1.5 (prefixes), bge-m3 and all-MiniLM-L6-v2 (256 word pieces), and the Ollama library pages for bge-m3 and nomic-embed-text. The cards are not copied here.

## Spend

OpenRouter spend for all embedding runs, probes and timing (from `cache/spend.json`): $0.1194. Question set 2 generation: $0.0329 (`make_questions_2.out`). Total $0.152. The spend guard in `embed.py` aborts above $2 and never allows $3.

## Files

- `results.json`: summary per set, paired bootstrap, decision rule, tiers, set 2 readout, cost and speed
- `results.md`: the tables in the post (generated by `report.py`)
- `terminal.txt`, `anchor_check.out`, `run_hosted.out`, `run_local.out`, `run_set2.out`, `bench_*.out`, `aggregate.out`, `diag.out`, `probe.out`: saved console output (two tracebacks removed from `run_hosted.out`)

## Chunk statistics

`chunk_stats.py` and its saved output `chunk_stats.out` give the corpus numbers quoted in the post: 130 pages (80 + 50), 1,389 chunks (1,167 + 222), 10.68 chunks per page, cl100k tokens per chunk median 260 and maximum 399.