# Reproduction package: measurement instrument

*Who May Read the Web? Measuring How Top Websites Admit, Refuse, and Invite AI Crawlers and Agents (October 2026)*, Max Song.

This folder holds the code that drew the sample and collected the data. It has no dependencies beyond Node.js itself. It was written and run with Node.js 26.8.1 on Linux (aarch64); nothing in it is specific to that platform.

The crawler it implements is described for site operators at <https://maxsong.me/papers/who-may-read-the-web/crawler/>. That page is the upper bound of what this code may request.

## Files

| File | Role |
| --- | --- |
| `config.mjs` | Every constant: the two User-Agent strings, limits, timeouts, the 26 probes and their headers, strata, sampling salt, storage layout |
| `codebook.json` | The frozen codebook (2026-10-04): 54 crawler tokens with a first-party and a community purpose label, robots.txt directives, probes with validator rules, HTTP signals, fingerprints with evidence levels, reachability codes, comparison sets |
| `sample.mjs` | Downloads Tranco list 647LX once, draws the stratified sample, writes `../data/sample.csv` and `../data/frame.json` |
| `robots.mjs` | robots.txt parser and matcher written against RFC 9309; site-level verdicts; non-REP records; fingerprints of vendor-written text. Pure functions |
| `validators.mjs` | Generic rejection rules, two-tier validators per file, soft-404 detection, reachability coding, home-page signal extraction. Pure functions |
| `collect.mjs` | The collector, and the reader for the data it writes |
| `SCHEMA.md` | Every field of the collected records, with the rule behind it |
| `tools/freeze-codebook.mjs` | Regenerates `codebook.json` from the research draft and the code |
| `tools/gen-adversarial-corpus.mjs` | Writes a deterministic corpus of 649 hostile robots.txt files |
| `tools/differential.mjs` | Compares `robots.mjs` with Google's open-source parser, cell by cell |
| `tools/export-robots-bodies.mjs` | Extracts the collected robots.txt bodies so that the comparison can be repeated on real files |
| `tools/fingerprint-review.mjs` | Builds the hand-check bundles for unverified fingerprints: every matching collected file, or a random 20 drawn with a recorded seed |
| `tools/reference/` | Build script and batch front end for the reference parser (verification only) |

## Reproducing the sample

```sh
node sample.mjs --data-dir /path/to/data --check
```

This downloads `https://tranco-list.eu/download/647LX/1000000` into `/path/to/data/tranco/` (22.6 MB, once), recomputes the sample and requires it to equal `../data/sample.csv` byte for byte. `../data/frame.json` records the list's sha256, the download time, the strata and the salt. `--dry-run` shows what would happen; `--offline` forbids the download.

The sample: ranks 1–1,000 in full (1,000 domains); 3,000 domains each from ranks 1,001–10,000, 10,001–100,000 and 100,001–1,000,000. Within a stratum the domains are ordered by `sha256(salt + ':' + domain)` and the first *n* are taken.

## Collecting

Nothing is sent until you ask for it twice: a run that can reach real sites refuses to start without `--confirm-live-run`.

```sh
# 1. What would be requested? No network access at all.
node collect.mjs --dry-run --pilot 5

# 2. A pilot: the first 50 domains of every stratum.
node collect.mjs --data-dir /path/to/data --pilot 50 --confirm-live-run \
     --vantage-note "where this runs from: provider, country, IPv4 only"

# 3. The full run. --resume continues a data directory without contacting a finished site again.
node collect.mjs --data-dir /path/to/data --resume --confirm-live-run --vantage-note "..."
```

The full run continues the pilot's data directory, so the pilot's sites are not contacted a second time; their records are the ones the pilot wrote.

Other options: `--only-stratum S1,S2`, `--limit N`, `--concurrency N` (default 100 sites in flight; one site is always strictly sequential), `--exclude-file FILE`. `node collect.mjs --help` lists everything, including the options that exist only for tests.

**Stopping.** Press Ctrl-C once: sites in flight finish and are recorded, no new site is started, and `--resume` continues later. A second Ctrl-C aborts. Sites that an aborted or crashed run had started are *not* requested again by `--resume`; they are listed in the manifest. `--recrawl-interrupted` requests them again, which can put up to 26 more URLs on a site that already received some.

**How long it takes.** Each site receives its requests one after another with at least one second between them, so a site that answers everything takes about half a minute and a site that redirects every path takes about twice that. At the default concurrency this puts the full sample in the range of one to a few hours; hosts that time out add to it. These are estimates from the limits, not measurements.

**What it writes.** See `SCHEMA.md`. Raw data stays outside the repository.

## Conduct

The collector:

- sends only `GET`, never a cookie, never credentials, never a `Referer`; runs no JavaScript; does not attempt challenges;
- identifies itself as `WhoMayReadTheWeb-Research/1.0 (+https://maxsong.me/papers/who-may-read-the-web/crawler/)` on every request but one: `/robots.txt` is fetched a second time with a common desktop-browser string, with otherwise identical headers, to compare the two answers. It never uses the name of any company's crawler, and refuses to build a request that would;
- requests at most 26 URLs per site; follows redirects by hand for at most five hops; retries a request at most once, after a network failure or after a 429 whose `Retry-After` is 30 seconds or less;
- reads robots.txt first and requests nothing it disallows for `WhoMayReadTheWeb-Research` or, failing a group for that token, for `*`. It also honours rules that are only valid under a lenient reading (a misspelled `Disallow`, a version suffix on the token). When robots.txt answers 5xx or cannot be reached it requests nothing else, as RFC 9309 prescribes;
- leaves a site alone when robots.txt and the home page are both refused, and when a site keeps answering 429;
- waits at least one second after a request to a host has ended before sending that host another one. All hosts of a sampled domain count as one host;
- finds a site at `https://<domain>`, then `https://www.<domain>`, then the same two over `http://`, moving to the next only when no connection could be made, so that at most one of them receives a request during this step;
- follows a redirect to another site only for robots.txt itself; for any other URL it follows redirects only within the sampled domain or to the host that served the robots.txt it read;
- never connects to loopback, private or reserved addresses, whatever a redirect says;
- cuts response bodies at 1,000,000 bytes, compressed or not, and keeps home pages only as extracted signals.

**Opt-out.** A site that adds

```
User-agent: WhoMayReadTheWeb-Research
Disallow: /
```

receives the two robots.txt requests and nothing else. A domain whose operator asks to be left out goes into the file given to `--exclude-file` (one domain per line, `#` for comments): it is never contacted, and its record must be removed from anything published.

## Checking the robots.txt parser

Earlier studies found home-grown robots.txt parsers to be a source of error, so `robots.mjs` is checked against Google's open-source parser ([google/robotstxt](https://github.com/google/robotstxt), commit `22b355ff`), which is not a dependency of anything here.

```sh
bash tools/reference/build.sh --work-dir /tmp/robots-reference        # needs git, cmake, a C++17 compiler
node tools/gen-adversarial-corpus.mjs --out /tmp/robots-corpus
node tools/differential.mjs --robots-dir /tmp/robots-corpus/files --tokens /tmp/robots-corpus/tokens.txt \
     --paths /tmp/robots-corpus/paths.txt --reference /tmp/robots-reference/build/robots_batch
```

`robots.mjs` has two interpretation profiles. `GOOGLE_PROFILE` reproduces the reference; `RFC_PROFILE`, the paper's primary reading, follows RFC 9309 where the two differ. The tool compares both with the reference on every (file, token, path) cell. On the 649-file corpus (1,037,102 cells):

- `GOOGLE_PROFILE`: 0 disagreements;
- `RFC_PROFILE`: 51,787 disagreeing cells, every one of them explained by one or more of twelve named differences between RFC 9309 and Google's implementation, none unexplained.

The twelve differences are the switches of the profile objects in `robots.mjs`: field names matched exactly rather than by prefix; no typo tolerance; no missing-colon tolerance; the whole `User-agent` value compared rather than its leading `[A-Za-z_-]` run; `*` followed by text is not the global group; no `index.html` special case; percent-encoded unreserved characters decoded on both sides; `/robots.txt` implicitly allowed; and three properties of the C implementation (a NUL byte ends a line, lines are cut at 16,663 bytes, a partial byte order mark is skipped). How the HTTP status of the robots.txt fetch is read (4xx, 429, 5xx) is a property of the fetcher, not of the parser, and is recorded separately (`SCHEMA.md` section 5.3).

To repeat the comparison on collected files: `node tools/export-robots-bodies.mjs --data-dir /path/to/data --out /tmp/robots-real`, then run `differential.mjs` on `/tmp/robots-real`. The report of that comparison on the files collected for the paper (5,338 distinct bodies, every differing cell attributed to a documented difference between the two implementations, none unattributed) is published as `../data/differential-report.json`; the two paths in it are written relative to `<data-dir>`, the directory that held the reference binary and the exported bodies.

## Rebuilding the codebook

`codebook.json` is generated: token table, directive notes, HTTP signals and comparison sets come from the research-phase draft; probes and limits from `config.mjs`; fingerprints, validator rules and vocabularies from `robots.mjs` and `validators.mjs`.

```sh
node tools/freeze-codebook.mjs --draft /path/to/codebook-draft.json --check
```

Fingerprints and edge signatures carry an `evidence` level. Those marked `r2-verified` were checked against a source during the research phase. Those marked `vendor-documented` were either described in vendor documentation during the research phase or raised after the collection under the protocol's section 10 rule; a raised entry records its `sourceUrl`, `verifiedOn` and `handCheck` (the log is published beside the paper as `../implementation/fingerprint-verification.md`); the one entry that was raised and withdrawn the same day (`cloudfront-request-blocked`, confirmed by both reviewers on 16 of 20 files, below the 19 required) keeps its `sourceUrl` and `handCheck` for the record and stays `implementer-recall`, without `verifiedOn`. Entries still marked `implementer-recall` or `needs-capture` are recorded so that the data can be grouped; no count built on them enters a measurement under a vendor's name, and the paper reports their matches as unidentified (it may still name the vendor in a sensitivity note, as it does for CloudFront).

The protocol's rule for raising an entry to verified (section 10) has two parts: a cited source for the text, and a hand check of the collected files that matched, all of them when there are 20 or fewer and otherwise a random 20 with at least 19 confirmed. The bundles for that check come from:

```sh
node tools/fingerprint-review.mjs --data-dir /path/to/data --out /tmp/fingerprint-review
```

One JSON file per unverified fingerprint holds the drawn matches (the robots.txt lines around the match, or the status, headers and saved head of an edge page) and the draw rule; `index.json` records the seed, so the same data gives the same draw. The outcome of the check is logged in the protocol's deviations section; an entry that passes gets its `evidence` and `sourceUrl` updated in `robots.mjs` or `validators.mjs`, after which `codebook.json` is regenerated.

## Tests

The instrument's tests are part of the project repository, outside this folder (`test/research/wmrw-*.test.mjs`, run with Vitest). They exercise the collector only against a fixture server on `127.0.0.1`; no test contacts a real site. Two of them need the reference binary (`WMRW_ROBOTS_REFERENCE=/path/to/robots_batch`) and one needs the cached Tranco list (`WMRW_DATA_DIR=/path/to/data`); without those they are skipped.

## Limits of the instrument

`SCHEMA.md` section 9 lists what the data cannot show. In short: one vantage point and one pass; the browser identity is a header string sent by the same client, not a browser; only the registrable domain is probed; IPv4 is preferred; only root-level paths are probed.

## Analysis

The rest of this folder turns the collected records into the numbers, tables and figures of the paper. It follows the pre-registered protocol (`docs/research/who-may-read-the-web/protocol.md` in the project repository), which was frozen before any site was contacted. Like the instrument, it needs nothing but Node.js.

| File | Role |
| --- | --- |
| `categorize.mjs` | Labels the sampled domains from open sources (Wikidata, domain-name rules, the Public Suffix List); writes `../data/categories.csv` |
| `derive.mjs` | Turns the records into one row per sampled domain: `sites.csv` and `sites.meta.json`. Every rule of the protocol that turns an observation into a variable lives here |
| `analyze.mjs` | Counts that table: intervals, stratum-weighted estimates, the six pre-specified tests. Writes `results.json`, `numbers.json`, `tables.json` |
| `figures.mjs` | Draws the figures from `results.json` alone: `figures/*.svg` and `figures/figures.json` |
| `portable-math.mjs` | `exp`, `log` and `log1p` written with basic arithmetic only (the fdlibm algorithms), so that `analyze.mjs` gives the same bits on every platform |
| `synthetic.mjs` | Writes an invented data directory in which the true value of every quantity is known. For tests and for trying the pipeline; never a measurement |

### From records to paper

```sh
D=/path/to/data            # the collector's data directory
T=/path/to/table           # where the per-site table goes

node derive.mjs  --data-dir $D --out-dir $T        # sites.csv, sites.meta.json
node analyze.mjs --sites $T/sites.csv --comparators ../data/comparators.json   # results.json, numbers.json, tables.json, into the paper's folder (..); the published files were made with --comparators
node figures.mjs --results ../results.json         # ../figures/
```

Each script takes `--dry-run` (compute and report, write nothing) and `--check` (recompute and require the files on disk to be byte-identical; exit status 1 when they are not, with the files that differ named on standard error). The output is a pure function of the inputs: there is no clock, no randomness and no network access in any of the three. It is also the same on every platform. ECMAScript lets `Math.exp`, `Math.log` and their kin return any close approximation, and engines differ in the last bit (for `Math.log` of a probability, V8 on arm64 and on x64 disagree on about one input in sixty); since an exact interval is found by bisection, one bit can move its end point. `analyze.mjs` therefore takes its logarithms and exponentials from `portable-math.mjs`, whose every step ECMAScript fixes exactly.

`analyze.mjs --data-dir $D` computes the same three files straight from the records. The two routes give byte-identical output, so the published table is enough to reproduce every number. `--out-dir` sends the files elsewhere. `--comparators FILE` adds same-month figures of other trackers to one table; the file is `{ "schema": "wmrw-comparators/1", "rows": [{ "source", "dataDate", "frame", "rule", "denominator", "retrievedAt", "tokens": { "GPTBot": { "count": 344 } } }] }` and is filled by hand from the trackers' own pages.

For the full sample, `derive.mjs` and `analyze.mjs` together take a few seconds and under 1 GB of memory (measured on synthetic data of the same size).

### What comes out

| File | Content |
| --- | --- |
| `sites.csv` | One row per sampled domain that may be published, about 200 columns: record state, outcome of both robots.txt requests, the verdict lists of the file as served and with intermediary-written blocks removed, authorship class, signals, tier of every probed file, home-page signals, conflicts. The columns are documented in `sites.meta.json` |
| `sites.meta.json` | Columns with their meaning, the table's sha256, the number of sampled domains per stratum in each record state, collection dates, hashes of the inputs |
| `results.json` | Every measure in every scope with its counts, the six tests with the counts behind them, the within-operator comparisons, and the definitions in use |
| `numbers.json` | A flat map `key → { text, value, num, den, ci, note }`: the only place the manuscript takes a number from. A key is `<measure>.<scope>`; scopes are `s1`–`s4` (one rank stratum), `top10k`, `top100k`, `top1m` (stratum-weighted) and `sample` (the whole sample, unweighted). The protocol's Appendix C lists the measure families |
| `tables.json` | `id → { caption, markdown }`, the tables of the paper |
| `figures/figures.json` | `[{ id, file, width, height, alt, caption }]`, with one SVG file per figure |

A domain whose operator asked to be left out has no row in `sites.csv`; `sites.meta.json` keeps only how many such domains each stratum has, and they stay in the denominator "all sampled domains". For a request that arrives after collection, put the domain in a file (one per line) and pass `--exclude-file FILE` to `derive.mjs`.

### Synthetic data

```sh
node synthetic.mjs --out-dir /tmp/wmrw-synthetic --units 1      # 1,000 invented sites; --units 10 gives 10,000
node analyze.mjs --data-dir /tmp/wmrw-synthetic --sample /tmp/wmrw-synthetic/sample.csv \
     --frame /tmp/wmrw-synthetic/frame.json --categories /tmp/wmrw-synthetic/categories.csv --out-dir /tmp/wmrw-out
node figures.mjs --results /tmp/wmrw-out/results.json
```

The invented sites live under the reserved name `.example`; no request is sent anywhere. Each kind of site states by hand what the analysis must conclude about it (`truth.json`), and the tests require every number to be recovered exactly. Output computed from synthetic or test data says so in every note, caption and alternative text, each figure carries a band across its top, and `derive.mjs`, `analyze.mjs` and `figures.mjs` refuse to write such output into the paper's folder.

### Categories

`../data/categories.csv` was computed once, on 2026-10-05, and is frozen with the protocol. `../data/category-sources/` holds everything it was computed from: the five Wikidata query results with the query text and time, the Public Suffix List as downloaded, and a manifest with their hashes.

```sh
node categorize.mjs --check        # recompute from the stored sources and compare, byte for byte; no network
node categorize.mjs --refresh --dry-run    # what a new retrieval would fetch and write
```

`--refresh` without `--dry-run` queries Wikidata and downloads the Public Suffix List again. Wikidata changes daily, so a refreshed file will differ from the frozen one; the frozen file is the one the paper uses.

A Wikidata website labels a sampled domain only when it is that domain's root (host equal to the domain or to `www.` plus the domain, path `/`). Without that rule the largest hosting platforms would be labelled as newspapers, universities and ministries at once. Curlie, the open directory named in the plan as a cross-check, could not be retrieved (HTTP 504 and timeouts on three attempts on 2026-10-05) and is not used; the manifest records this.

### Tests of the analysis

`test/research/wmrw-categorize.test.mjs`, `wmrw-derive.test.mjs`, `wmrw-derive-fidelity.test.mjs`, `wmrw-analyze.test.mjs` and `wmrw-figures.test.mjs` in the project repository. They check the interval and test procedures against worked examples from the literature, recover the known answers of the synthetic data number by number, compare the synthetic records with the ones the real collector writes when the same invented sites are served from `127.0.0.1`, and check that every number key the protocol cites is produced and that the inputs the protocol froze are unchanged. `wmrw-published-outputs.test.mjs` is the drift gate on the published files: `analyze.mjs --check` and `figures.mjs --check` against the committed per-site table, the built page against the manuscript and `numbers.json`, and the numbers quoted in the site's publication list against their keys. `wmrw-portable-math.test.mjs` pins the bits that `portable-math.mjs` and two exact intervals must give. Its inputs include ones on which V8 for arm64 returns other bits than the fdlibm algorithms, so a return to the platform's own `Math` functions fails there at once rather than only when another machine checks the published files. None of them uses the network.

### Limits of the analysis

- Authorship classes for platform default texts stay empty until the fingerprints behind them are verified against collected files (protocol, section 10); until then such files are counted as `unverified-template`.
- A conflict between an invitation file and a rule in the `*` group cannot be observed: the collector obeys that rule and never asks for the file.
- Category labels cover about a tenth of the sample and contain errors; news, government and education are reported, retail and community only described.
- The stratum-weighted estimates assume the sample as drawn. Domains that could not be collected stay in the denominators; nothing is imputed.