# Fingerprint verification log

*Who May Read the Web?* · protocol section 10 · carried out 2026-10-09/10 (UTC), after the collection of all 10,000 sites and before the final analysis was run.

The rule being applied (protocol §10, confirmed at freezing as default 4 of §15): an unverified fingerprint or edge signature is raised to verified only when **(a)** a source for the text is cited (the vendor's documentation or the published default file) and **(b)** a hand check of the files collected in this study that matched it confirms it: all matching files when there are 20 or fewer, otherwise a random 20, with at least 19 of 20 confirmed. Fingerprints that stay unverified stay in `unverified-template` (robots.txt) or keep `evidence: implementer-recall` (edge signatures, whose matches then count as recognised only after this check).

## How the two parts were done

**(a) Sources.** A workflow of 16 agents (8 finders, 8 refuters; `wf_e216a296-d49`, 2026-10-09) looked for the vendor's own text for each of the 28 unverified entries: documentation, the vendor's source repository, or the default file as the vendor publishes it. A refuter then opened every cited link and checked that the quoted text is present, that the source is the vendor's own, and that the codebook pattern would match a file carrying it. The merged result is kept outside the repository (`~/research-data/who-may-read-the-web/fingerprint-sources.json`); the links judged usable are copied into `sourceUrl` of the raised entries.

**Interpretation applied to (a)**, recorded here because the rule's wording leaves it open: a source satisfies (a) when the vendor's own documentation or repository, or the default file as the vendor ships it, attests the identifying text, or, for an HTTP signature, the mechanism that produces the matched response (the header name and its values, the status, or the format of a reference or incident code on the vendor's own domain). A copy of the text in a third party's repository does not satisfy (a) on its own.

**(b) Hand check.** `code/tools/fingerprint-review.mjs` listed every collected file that matched each entry and drew the random 20 with the seed `who-may-read-the-web:<id>` (`index.json` records the seed and the counts). A second workflow of 16 agents (8 checkers, 8 refuters; `wf_461db484-635`) read every drawn sample: a checker judged each one confirmed, rejected or uncertain against the vendor text from (a) and the surrounding lines or headers; a refuter then re-read the same samples with the brief to overturn confirmations. A sample counts as confirmed only when both agreed. The author took the decisions below from those counts.

## Part (a): sources found

Match counts are over the complete records (10,000 sites). "Drawn" is the size of the hand-check sample.

| Entry | Matches (files / domains) | Drawn | Source under (a) | Notes |
| --- | --- | --- | --- | --- |
| `yoast-block` | 275 / 275 | 20 | **Yes.** Yoast SEO source, `src/presenters/robots-txt-presenter.php` (markers exact) | Markers unchanged since 19.7.1 (2022-09-20); 19.7.0 used a different name |
| `wordpress-core-rules` | 500 / 500 | 20 | **Yes.** WordPress core `wp-includes/functions.php`, `do_robots()` (exact for 4.4–7.0; 7.1 derives the path from `admin_url()`, same output by default) | The same two lines are reproduced by ClassicPress and by many hand-written files; a match means "WordPress's default rules", not "generated by core" |
| `wordpress-com-firehose` | 4 / 4 | 4 | **No.** Text found only in third-party snapshots (Protego test data); not in Automattic documentation or repositories | Stays unverified |
| `ghost-default` | 3 / 3 | 3 | **Yes.** Ghost's published default robots.txt (`Disallow: /ghost/` present; the file has more lines) | Single short rule; specificity rests on the hand check |
| `shopify-default` | 24 / 24 | 20 | **No under the strict reading.** The sentence is the first line of the `robots.txt.liquid` template Shopify writes into every store, seen only in merchant repositories synced by `shopify[bot]`; it is not in Shopify's documentation or repositories | Flagged for the author: a reading that accepts the deployed default file as "the published default file" would satisfy (a). Not adopted here |
| `squarespace-default` | 9 / 9 | 9 | **No.** No vendor text found | Stays unverified |
| `wix-auto-generated` | 13 / 13 | 13 | **No.** No vendor text found | Stays unverified |
| `blogger-default` | 22 / 22 | 20 | **No.** No vendor text found | Stays unverified |
| `drupal-default` | 191 / 191 | 20 | **Yes.** Drupal core `core/assets/scaffold/files/robots.txt` (comment unchanged since 2006) | The same comment ships in PrestaShop and Backdrop files; the class "platform template" is what is claimed, not the specific platform |
| `prestashop-generated` | 19 / 19 | 19 | **Yes.** PrestaShop `classes/Tools.php`, `generateRobotsFile()` | 1.6.1.x spelt "automaticaly"; such files do not match |
| `joomla-default` | 19 / 19 | 19 | **Yes.** Joomla `robots.txt.dist` | |
| `rails-default` | 26 / 26 | 20 | **Yes.** Rails application template `public/robots.txt` | Reused by Discourse and other Rails applications; same remark as for Drupal |
| `cf-other-block` | 0 / 0 | 0 | No third label documented by Cloudflare | No match; stays as the catch-all it was |
| `knownagents-per-agent` | 238 / 238 | 20 | **Layout documented, threshold not.** Known Agents' REST documentation shows the per-agent layout (one `User-agent`, one `Disallow: /`, repeated); the `>= 20` threshold is the codebook's own, and the documented example is truncated, so the refuter judged that the pattern does not match the documented text | Whether a match is Known Agents output is decided by vendor markers in the hand check |
| `rsl-collective-notice` | 0 / 0 | 0 | **Yes.** RSL's own robots.txt | No match; cannot be hand-checked, stays unverified |
| `cloudflare-access-denied-title` | 0 / 0 | 0 | Blog example only | No match |
| `cloudflare-error-code-text` | 0 / 0 | 0 | Logpush documentation shows `error code: 1020` as a plain-text answer | No match |
| `vercel-challenge-header` | 13 / 7 | 13 | **Yes.** Vercel's `vercel/workflow` repository documents `x-vercel-mitigated: challenge \| deny` as set by the Vercel firewall | Not in the product documentation (checked: 0 occurrences in `vercel.com/docs/llms-full.txt`) |
| `vercel-deny-header` | 34 / 2 | 20 | **Yes.** Same source (the refuter's objection was to a quotation that spanned two files; the header value `deny` is in the cited comment) | |
| `aws-waf-action-header` | 146 / 20 | 20 | **Yes.** AWS WAF developer guide and API reference: `x-amzn-waf-action` with `captcha` or `challenge` | Sent only when the token is missing or invalid |
| `cloudfront-request-blocked` | 108 / 34 | 20 | **Yes.** AWS Knowledge Center (AWS official, updated 2026-03-17) documents the 403 "The request could not be satisfied. Request Blocked" page; a sibling article documents the "Bad Request" variant; the page template with "Generated by cloudfront" is in AWS's own SDK repository. Verified by the author in a browser on 2026-10-10 | The signature therefore means "a 403 generated by CloudFront", WAF block or misconfiguration alike; both are the edge refusing |
| `akamai-access-denied` | 370 / 130 | 20 | **Partly.** Akamai documents the error reference code format (`Reference #18.…`) and that the standard deny action returns 403; the title "Access Denied" and the `errors.edgesuite.net` link are not quoted in documentation | Accepted under the interpretation above (documented mechanism on the vendor's own domain); the hand check decides |
| `imperva-incident` | 130 / 20 | 20 | **Partly.** Imperva (Thales) documents that its error page shows an incident ID; the words "Incapsula incident ID" and the `_Incapsula_Resource` script are not quoted in current documentation (Incapsula is the product's former name) | Same treatment as Akamai |
| `datadome-captcha` | 226 / 29 | 20 | **Yes.** DataDome documentation: challenges are served as HTTP 403 and load `captcha-delivery.com`, DataDome's own domain | |
| `perimeterx-captcha` | 55 / 11 | 20 | **Yes.** HUMAN Security (PerimeterX) enforcer repositories: block template with `px-captcha` and `_pxAppId`, served with 403 | The same template serves hard blocks and captcha challenges, so `edge-challenge` here includes hard blocks; sites with the HUMAN sensor can also carry `_pxAppId` on their own 403 pages |
| `sucuri-firewall` | 0 / 0 | 0 | Third-party only | No match |
| `f5-request-rejected` | 5 / 3 | 5 | **Yes.** F5 knowledge base K92414462 quotes the default text; default policy in F5's own repository | F5's default page answers 200, which this study's reachability coding would not see as a refusal |
| `anubis-challenge` | 1 / 1 | 1 | **Yes.** TecharoHQ/anubis source (`making_sure_not_bot` title) | Localised titles in 30 languages are not matched |

## Part (b): hand check

Workflow `wf_461db484-635` (first run 2026-10-09 22:09 PDT, interrupted; completed on resume 2026-10-10 10:10–10:54 PDT; 16 agents, 366 drawn files). "Checker" is confirmed / rejected / uncertain; "Both" is the number the refuter also confirmed, which is the count the rule is applied to. "Raised" means (a) and (b) both hold under the interpretations recorded in this file.

| Entry | Drawn | Checker | Both | Outcome | What the check showed |
| --- | --- | --- | --- | --- | --- |
| `yoast-block` | 20 | 20 / 0 / 0 | 20 | **Raised** | Markers are Yoast's in all 20. Two are physical files edited by hand after Yoast wrote the block (CRLF, extra lines); the block is still Yoast's text. Known Agents uses the same ruler layout with its own markers; no cross-match |
| `wordpress-core-rules` | 20 | 7 / 0 / 13 | 20 carry the rules; 5 are core output as served | **Raised**, description reworded | The 13 "uncertain" are hand-written or plugin-managed files that kept the two rules (CRLF, re-ordered, extra rules, Rank Math headers). Class 4 of §5.6 means "carries a platform's default text", so the question is whether the file carries WordPress's default rules: all 20 do. Under a "generated by core as served" reading the count is 5 of 20 and the fingerprint would stay unverified |
| `wordpress-com-firehose` | 4 | 4 / 0 / 0 | 4 | Stays (no source) | |
| `ghost-default` | 3 | 2 / 0 / 1 | 2 | Stays | One file has only the `/ghost/` line, a sitemap on another host and `Allow: /`: not Ghost's text. With 20 or fewer files every file must be confirmed |
| `shopify-default` | 20 | 20 / 0 / 0 | 19 | Stays (no source under the strict reading) | All 20 are served by Shopify (`powered-by`, `RobotsController` ETag). The refuter's one "uncertain" is a 7-line file whose only Shopify text is the first line. (b) would pass at 19 of 20; the source question is the author's |
| `squarespace-default` | 9 | 9 / 0 / 0 | 9 | Stays (no source) | All 9 served by Squarespace; the template lists 26 AI tokens by default, in the same group as `*` |
| `wix-auto-generated` | 13 | 13 / 0 / 0 | 13 | Stays (no source) | All 13 carry `x-wix-request-id`; the template carries a PetalBot disallow and crawl-delays by default |
| `blogger-default` | 20 | 3 / 17 / 0 | 3 | Stays | 17 are large sites' hand-written files that happen to have a Mediapartners-Google group and `Disallow: /search` |
| `drupal-default` | 20 | 20 / 0 / 0 | 20 | **Raised** | Over all 191 matching domains the refuter counted 23 PrestaShop files and 4 comment-only files (about 14%): the class "platform template" is right, the platform label is not always Drupal |
| `prestashop-generated` | 19 | 19 / 0 / 0 | 19 | **Raised** | The source is the generator code in PrestaShop's repository, accepted under the interpretation of (a) above |
| `joomla-default` | 19 | 19 / 0 / 0 | 19 | **Raised** | |
| `rails-default` | 20 | 2 / 0 / 18 | 2 | Stays | The Rails template has no directive, so a file that is only the template is `no-directives` anyway; the fingerprint only reclassifies files whose directives the site wrote. Refuter: 5 confirmed, 7 rejected, 8 uncertain |
| `knownagents-per-agent` | 20 | 0 / 14 / 6 | 0 | Stays | None of the 20 carries a Known Agents marker; most are circulated hand-made lists. Of the 211 distinct matching bodies in the corpus, 5 are the vendor's output. The layout heuristic does not identify the vendor |
| `vercel-challenge-header` | 13 | 13 / 0 / 0 | 13 | **Raised** | All 429 with the Security Checkpoint page. Six sites were challenged on their very first request (browser identity first), so the challenge is not a reaction to this collector's volume. The page carries per-request tokens, so the two identities' bodies differ (`robotsPair.sameBody` is false for these 6 sites) |
| `vercel-deny-header` | 20 | 20 / 0 / 0 | 20 | **Raised** | All 34 matches come from 2 domains with path-based rules: `/.well-known/*` and the random control paths refused, robots.txt and home page served to both identities |
| `aws-waf-action-header` | 20 | 20 / 0 / 0 | 20 | **Raised** | Judged on headers, status 202 and length: 14 bodies are empty by design and 6 home pages are not stored. Three sites challenged every path after a few requests, so the trigger is request count, not path |
| `cloudfront-request-blocked` | 20 | 20 / 0 / 0 | 16 | **Stays** (raised on 2026-10-10, withdrawn the same day after the closing review: decision 3) | The 4 disputed files: 3 are the "Bad request." variant (2 domains in the draw; 9 of 108 matches overall, on 3 domains), a CloudFront 403 from a distribution misconfiguration, refused to both identities alike; the fourth is a "Request blocked." page generated by CloudFront but relayed by an Akamai edge in front of it (1 domain, 5 of 108 matches), which the refuter would attribute to Akamai. Under the interpretation recorded in Part (a) the signature means "a 403 generated by CloudFront", which all 20 are; under a "WAF block only" reading the count is 17 of 20 (16 when the relayed page is attributed to the relaying edge) and the signature would stay unverified. (Corrected 2026-10-10 after the closing review: this entry had called all 4 disputed files the "Bad request." variant.) Six home-page samples have no stored body; both agents rebuilt the template with the response's `x-amz-cf-id` and matched the recorded sha256 |
| `akamai-access-denied` | 20 | 20 / 0 / 0 | 20 | **Raised** | One home-page sample reconstructed from `akamai-grn`. In the data the match is carried by the entity-encoded `Reference&#32;&#35;` alone; the literal `errors.edgesuite.net` never occurs unencoded |
| `imperva-incident` | 20 | 6 / 14 / 0 | 6 | Stays | See the defect above |
| `datadome-captcha` | 20 | 20 / 0 / 0 | 20 | **Raised** | Three variants (`t=fe` captcha, `t=bv` apparently a hard block, `rt=i` device check), all counted as `edge-challenge` |
| `perimeterx-captcha` | 20 | 18 / 0 / 2 | 18 | Stays | The 2 unverifiable files are one site's responses whose markers lie beyond the stored 2 kB head; 18 of 20 is below the threshold. Re-fetching would need new requests |
| `f5-request-rejected` | 5 | 5 / 0 / 0 | 5 | **Raised** | |
| `anubis-challenge` | 1 | 1 / 0 / 0 | 1 | **Raised** | |
| 5 entries with no match | 0 | — | — | Stay | `cf-other-block`, `rsl-collective-notice`, `cloudflare-access-denied-title`, `cloudflare-error-code-text`, `sucuri-firewall` |

Side findings of the refuters that bear on the analysis, not on the rule: the Squarespace and Wix templates name AI tokens by default (Squarespace: 26 tokens in the same group as `*`, mostly path-level rules; Wix: PetalBot fully disallowed), so 22 sites carry platform-default naming and 14 of them (13 Wix, 1 Squarespace) count as a full refusal of a named token under the paper's definition; the raised `vercel-challenge-header` sites inflate `pair.different-body` (6 sites) because the challenge page differs per request; `servedBy` under-labels Akamai when `ak_p` is not the first value of a multi-valued `server-timing` header (1,721 responses, 138 domains; a hint, not a measure).

### Defect found by the check: `imperva-incident` matches ordinary pages

The checker for `imperva-incident` found that the signature's second alternative, the string `_Incapsula_Resource`, is the path of a script that Imperva injects into pages it proxies, so that any page under 16 kB that passes through Imperva matches, whatever its status; and that a robots.txt that lists `Disallow: /_Incapsula_Resource` matches too. The signature does not look at the status code or the content type. Counted over the complete records (read only; `imperva-impact.mjs`, kept outside the repository):

| Matches of `imperva-incident` | Responses |
| --- | --- |
| All | 130 responses, 20 domains |
| 404 `text/html` | 71 |
| 200 `text/html` | 27 |
| 403 `text/html` | 22 |
| 200 `text/plain` | 10 |

The ten `200 text/plain` responses are the robots.txt of six sites (research and browser identity), each a real file that mentions the Imperva path: their robots.txt is recorded as `edge-block`, its rules were not parsed, and `validation.valid` is false. Two further sites answered 404 for robots.txt with an Imperva-proxied page and are recorded as `edge-block` instead of `absent`. Because a refused robots.txt triggers the refusal check, three of these eight sites were left alone after the home page was also classed as refused (two with a real robots.txt, one with a 404); their remaining probes were never requested.

The signature stays at `implementer-recall` (6 of 20 confirmed) and nothing was changed: the collector, the records and the analysis are as frozen. The decision whether to correct the signature and to recompute the reachability of these eight sites from their stored bodies (possible for all eight; the three stopped sites cannot be completed without new requests) is the author's, and is recorded under Decisions. The paper discloses the defect with these counts.

## Decisions

Taken by the author on 2026-10-10, before the final analysis was run, except decision 3, which was revised the same day after the closing review. Each one is mechanical under the rule once the interpretation is fixed; the interpretations are stated so that a reader can apply a different one.

1. **Raised to `vendor-documented`**: `yoast-block`, `wordpress-core-rules`, `drupal-default`, `prestashop-generated`, `joomla-default`; `vercel-challenge-header`, `vercel-deny-header`, `aws-waf-action-header`, `akamai-access-denied`, `datadome-captcha`, `f5-request-rejected`, `anubis-challenge`. Each entry records `sourceUrl`, `verifiedOn: 2026-10-10` and `handCheck`. (`cloudfront-request-blocked` was raised with them and withdrawn the same day; decision 3).
2. **Reading of `wordpress-core-rules`**: a match means the file carries WordPress's two default rules, the reading of class 4 in §5.6 ("carries a platform's default text"); the description says so. The paper describes `platform-template` as "carries a platform's default text", not "written by the platform".
3. **`cloudfront-request-blocked` stays unverified** (decided 2026-10-10 after the closing review; it replaces the raise first recorded here). The raise had read the signature as "a 403 generated by CloudFront", a reading written in Part (a) before the hand check, under which all 20 drawn files match. But Part (b)'s own rule counts a file as confirmed only when both reviewers agreed, and they agreed on 16 of 20, below the 19 the protocol requires: the four disputed files are 3 of the "Bad request." variant and 1 page relayed by another vendor's edge. The closing review called the raise an interpretation rather than the mechanical rule, and the author withdrew it. The entry keeps its `sourceUrl` and `handCheck` for the record at `implementer-recall`; its 108 matches on 34 domains count as refusals, not as refusals by a verified signature, and the paper reports the "Bad request." variant (9 of 108 matches, 3 of 34 domains) as a sensitivity note.
4. **Files of 20 or fewer**: every file must be confirmed by both agents. `ghost-default` (2 of 3) therefore stays unverified.
5. **Unverifiable files count as not confirmed**: `perimeterx-captcha` stays at 18 of 20. The author may re-fetch the two responses; that needs new requests and is not done here.
6. **No fingerprint is tightened after the results were seen**: `blogger-default` keeps its pattern although the check found a feature that would separate the 3 true files from the 17 false ones; the paper reports the false-positive rate instead.
7. **`imperva-incident` is disclosed, not corrected** (see the defect section and protocol §11). Whether to correct the signature and recompute the eight sites from stored bodies is left to the author.
8. **Shopify**: the deployed `robots.txt.liquid` starting template, seen only in merchant repositories synced by `shopify[bot]`, is not accepted as "the published default file"; `shopify-default` stays unverified (24 sites in `unverified-template`). Accepting it would move them to `platform-template`.
9. **Interpretation of (a)** for `akamai-access-denied` and `prestashop-generated`: a documented mechanism on the vendor's own domain, or generator code in the vendor's repository, attests the text.

Effect on the repository: `robots.mjs`, `validators.mjs`, `tools/freeze-codebook.mjs` (evidence-level wording), `codebook.json` (regenerated; sha256 `a49318ffe4ec17d4eb2204779c720d7deebe1fbaba76ebd09d70b7a14e766439` at the raise, `af4a50eb17261b04b58979344f611f141453af6a695af963770555139fabe31c` after the two hand-check strings were corrected and the CloudFront raise withdrawn on 2026-10-10), protocol §2 and §11, `SCHEMA.md`, `README.md`. Tests that pin the verified set at freezing are updated with the raise.