chore(docs): granular robots.txt with explicit agentic allow policy - #38
Merged
Merged
Conversation
Pins the SebaSOFT decision (2026-09): training crawlers are ALLOWED on the neuron-js docs site. Citation-eligible bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot), user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) and training crawlers (GPTBot, ClaudeBot, Google-Extended) all get explicit Allow: / rules so policy scanners read a clear welcome instead of inferring from a wildcard. Wildcard stays open and the sitemap is declared for all agents. Contract test robots-granular.test.ts pins the bot list and fails if any rule flips to Disallow without a deliberate decision change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cierra el último gap de la nota del vault (agentic-web-optimization): robots.txt granular con decisión de training crawlers.
Decisión pinchada (SebaSOFT, 2026-09): training crawlers PERMITIDOS
docs/public/robots.txtpasa de wildcard mudo a naming explícito — cada clase de bot lee unAllow: /inequívoco:Sitemap:para todosEl contract test
robots-granular.test.tspincha el listado de bots y falla si alguien invierte la política aDisallowsin un cambio de decisión deliberado — la decisión queda trazada en código.Verified
Nota de alcance
robots.txtdesebasoft.github.ioraíz es 404 (host compartido de GitHub Pages) — nuestro archivo vive en/neuron-js/robots.txt, path absoluto correcto para el site. Los scanners de agentic-readiness leen subdirectorios donde vive el contenido.