robots.txt
A file at the root of a site telling automated visitors which paths they may fetch — and, increasingly, which AI crawlers are allowed at all.
It controls fetching, not indexing, and the difference bites. A page blocked here can still appear in results, listed from the links pointing at it, because the crawler was never allowed to fetch the page and read the instruction that would have excluded it. Keeping something out of an index is a job for a directive on the page, not for this file.
It is a request that well-behaved clients honour, and the honouring is the whole mechanism — there is no enforcement in it. Blocking a crawler that ignores the file takes something with teeth, at the server or in front of it.
The AI clauses are the part that changes fastest. Training crawlers, retrieval crawlers that fetch a page because a user asked about it right now, and search crawlers are different clients under different names, and a site can allow one while refusing another — which makes this file an editorial statement as much as a technical one.