Research / 2026-11-10
AI crawler access at 87 small-business sites
We fetched robots.txt from 99 small-business websites across 20 industries and city searches. 87 responded. 79.3% allow every AI crawler we tested. 17.2% block training and search crawlers together. Only 3.4% separate the two.
Sample collected 10 November 2026. Raw data: CSV, 87 rows.
Why this question
AI answer engines cite pages they are allowed to read. If a site blocks the crawler that builds the citation index, it does not rank lower — it is absent from the answer entirely. This does not appear in analytics, because the visit never happens.
Large publishers discuss this openly and configure it deliberately. Small businesses rarely do. The question was whether they differ, and by how much.
Method
Sample. Domains were collected by searching 20 industry-and-city combinations (dental in Austin, HVAC in Charlotte, law in Denver, and so on) and taking the local-business results. Directory sites, review platforms, government domains and national chains were removed. 99 domains remained; 87 served a responsive robots.txt.
Crawlers tested. Six search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User) and ten training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, Diffbot, cohere-ai).
Matching. Groups were parsed following RFC 9309 semantics: consecutive User-agent lines share one rule block, a bot-specific group takes precedence over the wildcard, and Disallow: / blocks unless overridden by Allow: / in the same group.
What counted as blocking. A site is recorded as blocking a crawler only when that crawler is fully excluded site-wide. Partial path exclusions were not counted.
Results
| Behaviour | Sites | Share |
|---|---|---|
| Allow every AI crawler | 69 | 79.3% |
| Block training, allow AI search | 3 | 3.4% |
| Block training and AI search | 15 | 17.2% |
| Block AI search only | 0 | 0.0% |
16 sites (18.4%) served an llms.txt file at the site root. None of the sites without one returned a 404 from a custom page — every miss was a genuine absence.
Finding: it is all or nothing
The interesting number is not 17.2%. It is 3.4%.
Of the 18 sites that block training crawlers, 15 also block the crawlers that build AI citation indexes. Only three separate the two decisions. That is a different shape from what large publishers do.
Scout Radar, which reads robots.txt from the Tranco top 10,000, reports 20.2% blocking at least one training crawler and 9.1% blocking training while keeping AI search open. In other words, roughly 45% of top-site blockers make the distinction. In this sample it is 17% (3 of 18).
With 18 blocking sites the confidence interval is wide, so this should be read as a signal rather than a measurement. But the direction matches what the mechanics predict: large sites have someone whose job is to know the difference, and small sites copy a template or accept a default.
The practical consequence is that a small business which wanted to opt out of model training, and thought that is what it did, has also opted out of being cited. The two decisions are made with different user-agent strings in the same file, and nothing in the interface explains that.
By industry
| Industry | Sites | Block search | Block training | llms.txt |
|---|---|---|---|---|
| HVAC | 8 | 2 | 2 | 0 |
| Family law | 7 | 3 | 3 | 2 |
| Physiotherapy | 7 | 2 | 3 | 0 |
| Moving | 7 | 1 | 2 | 4 |
| Law (general) | 6 | 1 | 2 | 1 |
| Accounting | 6 | 2 | 2 | 2 |
| Roofing | 6 | 0 | 0 | 1 |
| Dental | 5 | 2 | 2 | 1 |
| Landscaping | 5 | 1 | 1 | 0 |
| Veterinary | 5 | 0 | 0 | 1 |
| Auto repair | 5 | 0 | 0 | 3 |
| Dentist (solo) | 4 | 0 | 0 | 0 |
| Electrician | 4 | 0 | 0 | 1 |
| Chiropractic | 4 | 1 | 1 | 0 |
| Optometry | 4 | 0 | 0 | 0 |
No industry is clean. Legal and dental practices, which handle more regulated content, block more often — but the cell sizes are four to seven sites, so nothing here supports a claim about a sector.
Limitations
- 87 sites is a small sample. At this size the margin of error on a share near 20% is roughly ±5 to ±8 points at 95% confidence. Differences smaller than that are not distinguishable from noise here.
- Not a random sample. Domains came from search results for industry-and-city queries. Sites that rank well enough to appear may be better maintained than average, which would understate the problem, not overstate it.
- Chains are included. Some results were franchise or multi-location brands rather than independent businesses. They were kept because the page is still a local-service site, but they are not representative of a single-location owner.
- One snapshot, one day. robots.txt changes. This is a reading of 10 November 2026, not a trend.
- robots.txt only. WAF rules, CDN bot management and server-side blocking are invisible to this method. A site recorded as allowed may still be challenged at the edge.
- Nothing here is extrapolated to "small business" as a category. These 87 sites are what they are. They do not establish a rate for any industry, region or market segment.
Raw data
Every row is in the CSV, including domains, industry label, robots.txt status, per-category block counts and whether an llms.txt was served. No scores, no derived indices — just the measurements.
Collected with a script that fetches robots.txt over HTTPS with a named user agent, parses groups per RFC 9309, and evaluates each of the 16 crawler tokens against the matching rules. The same logic runs in the free AI crawler checker.
How to cite
Sacred Lantern Studio (2026). "AI Crawler Access at 87 Small-Business Sites." https://sacredlantern.online/en/research/ai-crawler-access-2026 Data collected 10 November 2026.
Revision history
- 2026-11-10 — Published. 99 domains collected, 87 reachable, results as shown.
Author: Sacred Lantern Studio, Suzhou. Contact: SSSkairos@outlook.com. Comparison figures for the Tranco top 10,000 are from Scout Radar (AI Crawler Check), fetched 20 October 2026, and are cited as their measurement, not ours.