@visibilitymesh/crawler-check

Check whether AI answer-engine crawlers can access a URL, and read the basic page signals that decide whether an AI assistant can understand your page. Zero dependencies. Node 18+.

It answers a simple question every store owner now has: can ChatGPT, Claude, Perplexity, Google AI, and the other AI crawlers actually reach and read this page?

Install

npm install -g @visibilitymesh/crawler-check

Or run without installing:

npx @visibilitymesh/crawler-check https://example.com

Usage (CLI)

crawler-check https://example.com
crawler-check https://example.com/products/thing --path=/products/thing
crawler-check https://example.com --json

Example output:

AI crawler access for https://example.com/
registry v0.2.0 (reviewed 2026-08-13)  robots.txt: found

  allowed  GPTBot                OpenAI · training
  BLOCKED  PerplexityBot         Perplexity · indexing   <- allowing this helps you get cited
  allowed  Googlebot             Google · indexing
  ...

Page legibility
  status:          200
  title:           Example
  meta description: present
  h1:              present
  structured data: 1 block(s) [Product]
  llms.txt:        not found

Action: 1 recommended AI crawler is blocked in robots.txt: PerplexityBot.

Usage (library)

import { checkUrl } from '@visibilitymesh/crawler-check';

const report = await checkUrl('https://example.com');
console.log(report.summary.recommendedCrawlersBlocked); // e.g. ["PerplexityBot"]

checkUrl returns { url, origin, robotsTxt, aiCrawlers[], page, llmsTxt, summary, registryVersion }. Also exported: parseRobots(text), isAllowed(robotsText, userAgent, path), readPageSignals(html), registry.

What it checks

• robots.txt access for every crawler in the AI Crawler Registry: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Amazonbot, Applebot, CCBot, Meta and DuckDuckGo AI bots, and more.
• Page legibility signals: HTTP status, title, meta description, h1, meta robots (noindex), canonical, JSON-LD structured data blocks and their types, and whether an llms.txt exists.

Scope and honesty

This tool reports public signals only (robots.txt and page HTML). It does not score your store, guess at fixes, or send your data anywhere. It is deliberately small and readable.

For a full audit, automated remediation on your store, and before and after validation of AI discoverability, see Visibility Mesh.

Part of Visibility Mesh

This is one of the open tools behind Visibility Mesh, the AI commerce visibility platform for Shopify. Find the full audit, remediation, and validation product on the store:

• Website: https://visibilitymesh.com
• Shopify App Store: https://apps.shopify.com/visibility-mesh
• All open tools: https://github.com/VisibilityMesh
• Companion data: ai-crawler-registry

License

MIT. Maintained by Visibility Mesh.