Free Tool
Batch AI crawler access checker
Check up to 10 sites in one run: AI crawler access, llms.txt, sitemap and structured data, side by side. Export the table as CSV for a client report or competitor comparison.
Who runs a batch check
| Who | List | What to look for |
|---|---|---|
| SEO or GEO agency | Every client domain | Any client blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot |
| Competitor research | Your site and four rivals | Who lets AI search in while you block it |
| Multi-site brands | Each location or subdomain | Inconsistent rules across hosts — robots.txt is per host |
| Migration QA | Old and new hostnames | The new host lost the original rule groups |
| Monthly monitoring | Key properties | Score drops after a CDN or plugin change |
Building a useful list
- One host per line. www.example.com and shop.example.com have separate robots.txt files.
- Enter the address people actually link to. If the bare domain redirects to www, use www — that's the file crawlers read.
- Comparable sites read better. Five competitors in one market beat twenty unrelated domains.
- Over 10 means multiple batches. Run one first to check the output before continuing.
How this differs from the single check
The scoring is identical: 70 points for crawler access, 15 for AI infrastructure, 15 for technical readiness. The difference is that the batch version doesn't list all 16 crawlers individually — it gives counts and totals, which is what you need for comparison.
To see which specific crawler is blocked, run the single check on that domain.
Limits
- 10 domains per run. Not a technical limit but a latency trade-off — more would time out.
- Four requests per site. No page content, so content quality and structure aren't in the results.
- Four batch runs per minute. Prevents use as a load-testing tool.
- WAF blocks are invisible here. robots.txt may allow while a firewall rejects earlier.