The web was designed for documents to be fetched by machines. Crawlers were not an attack; they were the point. Search itself is a crawler. So the technology layer grew up assuming a "reader" might be software, and its politeness protocol, `robots.txt`, is a request, not a fence.

That inheritance is why bot defense in tech is retrofitted rather than native. We bolt on CAPTCHAs, fingerprinting, and rate limits precisely because the substrate never distinguished a person "stepping" through an experience from a crawler sweeping through documents.

• `robots.txt` is honored by the polite and ignored by the motivated.
• CAPTCHA is friction for humans and a training set for machines.
• Fingerprinting is an arms race that penalizes privacy-conscious real users.

As major publishers raise their crawling defenses to protect content from being harvested for AI training, the open web quietly contracts. The crawler-friendly web is being retrofitted into a stepper-only web, one access wall at a time.