# LanceDB tutorial: code and outputs

Run date: 2026-11-06 (container clock showed 2026-11-05 23:xx UTC). Everything ran in Docker. Nothing is installed on the host or in this repo.

Scripts (each `X.py` has its saved output in `X.out`, except the helpers):
- `Dockerfile`, `requirements.txt`, `pip-freeze.txt`: the image. `run.sh` runs a script in it, `s3_run.sh` runs it with the bucket env, `common.py` is the shared helper (connect, chunks, query embedding).
- `version_check.py` (`setup.out`), `load.py` (`load.out`), `files.sh` (`files.out`, `files_after_index.out`), `query.py`, `filters.py`, `ann_index.py`, `fts_hybrid.py`, `upsert.py`, `gotchas.py`.
- S3 part: `s3_up.sh` (RustFS plus bucket), `s3_load_query.py` (`s3_load_query.out`), `s3_ls.sh` (`s3_ls.out`), `concurrent_write.py`, `concurrent_check.py` and `concurrent_run.sh` (`concurrent.out`).
- `build_chunks.py` (builds `chunks.jsonl` from a crawl), `questions.json`, `cleanup.sh`. `pipgrep.out` is the pip freeze filter shown in the first screenshot.

Run order: `docker build -t kb-lancedb:0.41.0 .`, then with `DATA_DIR` set to a host folder that holds `chunks.jsonl`:
`./run.sh version_check.py`, `./run.sh load.py` (then `./files.sh`), `query.py`, `filters.py`, `ann_index.py`, `fts_hybrid.py`, `upsert.py`, `gotchas.py`.
S3 part: set `S3_ACCESS_KEY` and `S3_SECRET_KEY` (throwaway local values, never commit them), run `./s3_up.sh`, then `./s3_run.sh s3_load_query.py`, `./s3_ls.sh ls s3://kb-lance --recursive --human-readable`
and `./concurrent_run.sh s3_run.sh append 100 shared` / `./concurrent_run.sh s3_run.sh upsert 40 shared` (the control uses `run.sh` with `LANCE_URI=/data/kb_conc`). Clean up with `cleanup.sh`.

Versions: `lancedb` 0.41.0 (PyPI, uploaded 2026-10-29), `pyarrow` 26.0.1, `numpy` 2.4.6, Python 3.11.18 (`python:3.11-slim`), Docker 24.0.6.
Local S3-compatible server: `rustfs/rustfs:1.0.4` (digest sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2, arm64), bucket tools via `amazon/aws-cli` (digest sha256:0cea63506f28774b70b1a3dd1934f49bc202d9784dee23370acc81438e8d0292).
MinIO was the first choice, but `minio/minio`, `minio/mc` (Docker Hub) and `quay.io/minio/minio` could not be pulled on 2026-11-06. AWS S3 and LanceDB Cloud were not tested.
Full dependency list: `pip-freeze.txt`.

Input: `chunks.jsonl` is NOT included (3.4 MB with embeddings). It has 304 rows, sha256 7c62db9258ba7b33bc75bf3e0593b0a7ffc73d0b0fb11da68db0eb27ebe4e145,
the same file as the Chroma tutorial. `build_chunks.py` (copied unchanged from that tutorial) builds your own from a WebCrawlerAPI crawl (key in env `WEBCRAWLERAPI_API_KEY`).
Queries need Ollama with `nomic-embed-text`; from the container it is reached at `host.docker.internal:11434`. `questions.json` has the 6 smoke-test questions.

Saved outputs (`*.out`) are the real, untrimmed output of the run. The screenshots drop the `WARN` log lines and some repeated rows.
`concurrent.out` times are `epoch mod 1000` seconds. `files_after_index.out` was captured after `fts_hybrid.py` had built two FTS indexes on the main table.