Research crawler · October 2026
You saw WhoMayReadTheWeb-Research in your logs
It is a one-time research measurement, run by one person, that reads the public files in which a website tells crawlers and AI agents what they may do. It does not index your content, train a model, or come back on a schedule.
Who and why
I am Max Song, an independent developer. This measurement supports a self-published technical report, Who May Read the Web?, about how popular websites admit, refuse, and invite AI crawlers and agents. The work is not affiliated with a university or company and has not been through an institutional ethics review. It follows the usual conduct for network measurement: identify yourself, ask for little, ask once, and make it easy to say no.
How to recognise it
Every request carries this exact User-Agent, with one disclosed exception described below:
WhoMayReadTheWeb-Research/1.0 (+https://maxsong.me/contact/)The link in the string is the contact page, which was live throughout the run. This page was published together with the report.
The sample is 10,000 domains drawn from the Tranco list 647LX, stratified by rank. Your site was selected by rank alone.
What it requests
At most 26 URLs per site, each with a single GET, in one pass. No host receives more than one request per second. A small pilot runs first.
- Redirects are followed for up to five hops, and only within your own site. A redirect of
robots.txtis followed wherever it leads, as RFC 9309 describes. - Retries. A request that fails for a network reason, or is answered with
429, may be repeated once. ARetry-Afterof up to 30 seconds is waited out; a longer one, or a second429, ends the visit, apart from the secondrobots.txtrequest described below. - Finding the site. If no connection can be made to
https://example.com, the first request is tried athttps://www.example.comand then over plain HTTP. - Leaving when refused. If
robots.txtand the home page are both refused, nothing else is requested.
| What | Paths |
|---|---|
| Crawler policy | /robots.txt, twice (see the exception below) |
| Home page | /, twice: once as HTML, once with Accept: text/markdown |
| Files written for language models | /llms.txt, /llms-full.txt, /ai.txt |
| Agent and API discovery | /.well-known/api-catalog, agent-card.json, agent.json, mcp/server-card.json, mcp.json, ai-catalog.json, ard.json, agent-skills/index.json, agents.json, ai-plugin.json; and /auth.md |
| Authorisation metadata | /.well-known/oauth-protected-resource, oauth-authorization-server, openid-configuration |
| Rights and baseline | /.well-known/tdmrep.json, /.well-known/security.txt; and one licence document, only if your robots.txt links to it |
| Controls | Two random paths that should not exist, to tell a real file from a catch-all page |
It never sends POST, never submits a form, never logs in, never runs JavaScript, and never tries to pass a challenge or CAPTCHA. Response bodies are cut off at 1 MB.
The one exception. /robots.txt is requested a second time with the User-Agent of a common desktop browser. Comparing the two answers shows whether a site's crawler policy can be read by a crawler that says what it is. This crawler never uses the name of any company's bot.
It follows your robots.txt
Your robots.txt is read first. The crawler then follows the group addressed to WhoMayReadTheWeb-Research, or the * group if there is none, as RFC 9309 describes. A path you disallow is not requested; it is recorded as “not fetched, disallowed”.
How to opt out
- Before or during the run: add the two lines below to your
robots.txt. Nothing exceptrobots.txtitself will be requested. - At any time: send your domain through the contact page. I will remove its record from the published dataset and exclude it from any later run.
User-agent: WhoMayReadTheWeb-Research
Disallow: /What is kept and published
For each request: the status code, the redirect chain, the IP address of the server that answered, and response headers from a fixed list of about 80 names such as Content-Type, Server and Link. The body of a policy or discovery file is kept in full. For a response that turns out to be an HTML page, only a hash, the length and the first 2 kB are kept; for the home page, only the signals read from it. No cookies are sent or kept and no personal data is sought. The report will publish a per-domain dataset of parsed results together with the collection and analysis code, so that anyone can check the numbers. A domain is identified only by its name and Tranco rank, both already public.
Something looks wrong?
If this crawler asked for more than this page says, asked too quickly, or caused any trouble, please tell me through the contact page. Include a log line if you can. I will stop the run for your site and reply.
中文说明
这是一次性的研究测量，用于自发表技术报告《Who May Read the Web?》。它只读取网站公开的爬虫策略与智能体发现文件，每站最多请求 26 个地址（只用 GET，重定向最多跟 5 跳），同一主机每秒不超过 1 个请求，并遵守你的 robots.txt。不想被测量，可在 robots.txt 里对 WhoMayReadTheWeb-Research 写 Disallow: /，或通过联系页告知域名。