# Code and data for "Markdown chunking in Python"

Everything here produced the numbers in the post. Python 3.11, versions pinned in `requirements.txt` (`pip-freeze.txt` is the full venv).

## Layout

Scripts expect crawled pages at `../data/<site>/pages.jsonl` (one JSON object per line: `url`, `title`, `markdown`). `pages.jsonl` is not included, because it is a copy of other people's documentation. Recreate it with `crawl.py`.

## Run order

```bash
export WEBCRAWLERAPI_API_KEY=...   # crawl.py
export OPENROUTER_API_KEY=...      # make_questions.py, evaluate.py (embeddings)

python crawl.py fastapi && python crawl.py htmx   # writes ../data/<site>/pages.jsonl, ~$0.26 on pay-as-you-go
python chunkers.py --freeze                       # writes params.json (chunker parameters and a hash of chunkers.py)
python make_questions.py                          # writes eval_set_draft.json (not included)
# spot check by hand, then freeze: eval_set.json + eval_set.sha256
python evaluate.py > evaluate.out                 # embeds (cached in cache/emb.pkl), scores, bootstraps, writes results.json
python report.py                                  # results.md, terminal.txt
python analysis.py > analysis.out                 # paired counts and failure examples
python show_page.py > show_page.out               # one page cut four ways
python chart.py; python diagram.py                # figures
```

## Freeze record

| What | When (UTC) | Hash |
|---|---|---|
| Chunker parameters (`params-run1.json`): cl100k_base, cap 400, overlap 60, merge below 80, split on `#` to `###`, heading path prefix | 2026-11-01 11:57:47 | params sha256 `8bfdbb1a1453530cb5a1a8f99d81045334b3360d8428a0761ae0e239e51f19b6`, chunkers.py (now `chunkers_run1.py`) sha256 `f71b3329b7630e40f8118559b58b9345cbbd11f31946d7185c737f91c87ac6ec` |
| Eval set (`eval_set.json`, 93 questions) | 2026-11-01 11:59:14 | sha256 `f274e1a16a8f72e6c59aa5998e22aa196201417f8f1f028385e6135c81259b52` |

Params frozen at 11:57:47 UTC and the eval set at 11:59:14 UTC, both before the first evaluation. `evaluate.py` refuses to run if `eval_set.json` no longer matches `eval_set.sha256`. Before the freeze the chunkers ran on the crawled pages to check for bugs (position bookkeeping, a duplicated-text bug in LangChain's start index, a heading permalink in FastAPI headings). No retrieval score was computed before the freeze. Question generation ran before the params freeze; the eval set was frozen after the hand check.

## Separator fix after the first run

After the first evaluation a review found a bug: the markdown separators from `get_separators_for_language(Language.MARKDOWN)` are regexes, and without `is_separator_regex=True` LangChain escapes them, so the heading separators never matched. This affected chunker B and the recursive fallback inside D and E. The fix (one added argument in `chunkers.py`, new `params.json` written 2026-11-01 12:14:11 UTC, chunkers.py sha256 `903c72991407326cc6b580b2f34e96ceba5302e635f1c076bd8578ce48d85485`) changed nothing else: same cap, overlap, merge threshold, prefix format and the same eval set. B, D and E were rechunked and rescored (A and C do not use that splitter). The first-run files are kept as `*-run1.*` (`results-run1.json`, `results-run1.md`, `terminal-run1.txt`, `analysis-run1.out`, `evaluate-run1.out`, `show_page-run1.out`, `corpus_checks-run1.out`, `params-run1.json`, `chunkers_run1.py`). Every number in the post comes from the second run. The extra embedding spend was 198,467 tokens.

`bootstrap_extra.py` is a post-hoc addition: paired bootstrap for E against A, B and C, which `evaluate.py` did not include.

## Hand check

24 of the 94 drafted questions were read: `random.Random(1).shuffle` of the draft list, first 24. 22 kept, 1 rewritten (id 93), 1 dropped (id 92). The log is `hand_check.json`.

## Crawl record

| Site | Start URL | Whitelist | Pages | Cost | Finished (UTC) |
|---|---|---|---|---|---|
| FastAPI docs (MIT licensed, checked 2026-11-01) | https://fastapi.tiangolo.com/tutorial/ | `^https://fastapi\.tiangolo\.com/(tutorial\|advanced)/` | 80 | $0.16 | 2026-11-01 11:54 |
| htmx docs | https://htmx.org/docs/ | `^https://htmx\.org/(docs\|reference\|attributes\|examples)/` | 50 | $0.10 | 2026-11-01 11:55 |

robots.txt read 2026-11-01 before the crawls: FastAPI has comment lines only (no directives), htmx allows all. Both crawls used `respect_robots_txt: true` and `max_age: 0` (see `data/<site>/crawl_meta.json`, copied here as `crawl_meta_<site>.json`).

## Question model

`openai/gpt-4.1-mini` via OpenRouter, temperature 0.3, sees one whole page at a time. See `make_questions.out` for generated, dropped and kept counts.

## Files

- `results.json`: summary, bootstrap and one row per question and chunker (top-5 chunks with scores)
- `results.md`: the tables in the post
- `terminal.txt`, `evaluate.out`, `analysis.out`, `show_page.out`, `make_questions.out`: saved console output