Add robots.txt - #436
Add robots.txt#436eugenia-scandit wants to merge 9 commits into
Conversation
/robots.txt returns 404 today. A missing robots.txt is not neutral — crawlers
read it as "everything permitted" — so the site's policy was in force by
omission rather than stated. This writes it down and declares the sitemap.
Guides are untouched: every major stays open to every crawler. People still run
6.x and 7.x and must find their documentation, in search and through an
assistant they ask. The guides already ship one tree per major (/ for 8.x,
/7.6.14/, /6.28.11/).
The API reference is a separate site on the same host — Sphinx, not Docusaurus,
built and deployed by another pipeline. robots.txt is per-host, so this one file
governs both. Unlike the guides it ships a tree per MINOR version: six versioned
trees are live alongside the unversioned one. Each was verified by fetching ONE
real page from it (index.html is only a redirect stub, so probing it proves
nothing): that page is byte-identical at 38,575 bytes between the unversioned
tree and /8.6/, is ~38 KB in each of the frozen 8.x minors /8.5/ /8.4/ /8.3/,
and ~35 / ~32 KB in /7.6/ and /6.28/. Those are one-page measurements — no claim
is made about tree completeness. The current version always lives at the
unversioned path, so /8.6/ is a pure duplicate.
The rule is default-deny and self-maintaining rather than a version list that
goes stale:
Allow: /7.6/data-capture-sdk/
Allow: /6.28/data-capture-sdk/
Disallow: /*/data-capture-sdk/
A new minor is excluded with no edit, and its content is already served at the
unversioned path. A new major is also excluded with no edit — the Disallow's `*`
matches whatever the first path segment is, so /9.0/, /9.1/, … are each caught
individually as they appear; there is no "/9.x/" pattern involved. Keeping the
outgoing major's tree then costs one Allow line, decided once at the release.
The Allow lines are placed FIRST deliberately. Under RFC 9309 longest-match
precedence the order does not matter, because the longer Allow beats the shorter
Disallow either way. But this group deliberately excludes Googlebot and
Bingbot, and none of the six agents in it publishes a conformance statement, so
no RFC 9309 guarantee applies to them. A first-match-wins parser (historical
Nutch / crawler-commons behaviour, which is CCBot's lineage) would read a
leading Disallow and deny /7.6/ and /6.28/ outright, silently breaking the
requirement that every major stays reachable. Evaluated under both models, the
six cases that matter now agree:
path longest-match first-match-wins
/data-capture-sdk/ allow allow
/8.6/… deny deny
/7.6/… allow allow
/6.28/… allow allow
/9.0/… (future) deny deny
/sdks/… (guides) allow allow
With the Disallow first, first-match-wins denied /7.6/ and /6.28/ — the only two
rows that changed.
Bulk crawlers only, and the usual objection does not apply here. Blocking crawl
on an indexed URL can strand it, so the standard advice is to use
canonical/noindex instead — but the Sphinx pages carry no <meta name="robots">
and no canonical link at all, and adding them means changing the Sphinx build,
not this repository. robots.txt is the only lever reachable from here. It is
applied to bulk crawlers, where stranding is irrelevant because nothing is
ranking them; search-engine de-duplication is left to the Sphinx side rather
than half-done here.
No Crawl-delay. No requirement asks for rate limiting, only two of the six named
agents document support for it, and Amazonbot's own documentation says it does
not support the directive — so the directive would have thrown an inaccurate
claim into the file while throttling ClaudeBot and CCBot across the whole site,
including the guides the "reachable by assistants" requirement wants open.
The file states no page count for the API reference. The "~3,200" and "~3,900"
figures elsewhere in this repo both count pages that dropped out of the SEARCH
INDEX during a tag rename; neither is a census of sitemap-absent pages. What is
recorded instead is measured: the sitemap carries 1,508 guide URLs (618 root,
498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ URLs, so the API
reference is in no sitemap and is link-discoverable only, with three trees
link-reachable. Also noted: 21 retired version trees are still live and
unsitemapped — /7.6.3/–/7.6.13/ and /6.28.1/–/6.28.10/, roughly 450-500 pages
each — while /8.5.3/, /8.5.2/, /8.4.1/ and /8.3.1/ return 404.
Docusaurus copies static/ to the build root — verified live, where
static/img/logo-light.svg serves as /img/logo-light.svg. Deployment and cache
invalidation happen outside this repository, and a second pipeline writes the
Sphinx trees to the same host, so confirm https://docs.scandit.com/robots.txt
returns 200 after deploy and is not later clobbered by that pipeline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
moritzhartmeier
left a comment
moritzhartmeier
left a comment
There was a problem hiding this comment.
Verified the factual claims against the live site and the repo before reviewing, and they hold up: /robots.txt is 404 today; the sitemap has exactly 1,508 URLs (618 root, 498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ entries; the Allow: paths match the real link targets in versioned_docs; the retired trees are still serving 200; and the preview build confirms static/robots.txt reaches the build root.
The rule is correct and I found no functional bug. Disallow: /*/data-capture-sdk/ does not catch the unversioned tree (the pattern needs a second occurrence of the literal segment), and the Allow-first ordering genuinely matters for a first-match-wins parser.
Three comments below, all about the commentary rather than the directives.
| # both count pages that dropped out of the SEARCH INDEX during a tag rename. | ||
| # Neither is a census of sitemap-absent pages, so neither can be reused as one. | ||
| # | ||
| # Also live, and also in no sitemap: 21 retired version trees — /7.6.3/ through |
There was a problem hiding this comment.
This publicly advertises 21 stale doc trees the file does nothing about.
robots.txt is one of the most-fetched files on any host. This block hands every crawler a discovery list of outdated guide trees that nothing currently links to and no sitemap carries — the opposite of the de-duplication the rest of the file argues for. It also puts an internal deploy postmortem ("the deploy never deleted what the build stopped producing", "~3,200 pages left search and nothing failed") on a public URL.
Suggest moving the census and the tag-rename history to the PR description or to a repo-internal comment. docusaurus.config.ts already carries this narrative and is not served.
Same goes for the measurements further up: the byte counts (38,575), the URL counts and the 2026-09-07 build date are all correct today and all go wrong at the next release with nothing failing. Either drop them, or assert them in scripts/ alongside the existing search-tag checks if they are load-bearing.
| User-agent: meta-externalagent | ||
| Allow: /7.6/data-capture-sdk/ | ||
| Allow: /6.28/data-capture-sdk/ | ||
| Disallow: /*/data-capture-sdk/ |
There was a problem hiding this comment.
Inverted priority — worth a follow-up ticket before this merges.
This rule dedupes the API reference for bulk crawlers, where it barely bites: I checked, and nothing in docs/ or versioned_docs/ links to any /8.x/data-capture-sdk/ tree, and none of them are in the sitemap. They are unreachable already, so the Disallow is future-proofing more than active de-duplication.
Meanwhile the 21 retired guide trees stay fully open to Googlebot, and that is where duplicate outdated content actually costs something.
The file's own stranding argument says robots.txt is the wrong lever there, and I agree — but then the right output is a ticket for the deploy to 410 those trees, not a comment noting they exist. Nothing else in this PR acts on them.
There was a problem hiding this comment.
Agreed on all of it, and I've taken the comment as two separate asks: stop the file implying it handles this, and get the real fix owned somewhere.
What changed here. The census is gone from the published file entirely — along with the deploy postmortem, the tag-rename history, the byte counts and the build date (your first comment). So the file no longer notes the trees exist, which was the half-measure you're objecting to. The narrative moved into the plugin source in docusaurus.config.ts, which is not served, and it now records the gap explicitly: ~9,500 pages, still 200, still self-canonical, fully open to Googlebot, and not addressed by anything in this PR.
On the priority itself — you're right, and I want to be precise about how right. I confirmed your point rather than taking it on trust: nothing in docs/ or versioned_docs/ links any /8.x/data-capture-sdk/ tree, and none are in the sitemap. So the Disallow is future-proofing, not active de-duplication. The 21 retired guide trees are the live cost.
Why I did not act on them here. Not deferral — robots.txt is the wrong instrument, by this file's own argument. Those trees are already indexed, so a Disallow strands them: the crawler can no longer fetch the page to read a noindex or canonical that would resolve the duplicate, and the URL can persist in the index with no snippet. Blocking crawl would make the situation harder to fix, not better.
The right output is a deploy change, and it needs a ticket:
410 Gonefor/7.6.3/–/7.6.13/and/6.28.1/–/6.28.10/. Not404:410is treated as a deliberate, permanent removal and de-indexes faster.- Or, if any of them must keep serving,
301to the current tree of that major (/7.6.14/,/6.28.11/) so the equity consolidates instead of being dropped. - Either way the underlying cause is the same — the deploy copies what the build produces and never removes what it stopped producing — so it is worth fixing as a deploy-prune step rather than as a one-off cleanup, or the next freeze recreates the problem.
I can't open that ticket from here (I don't know which tracker owns the deploy pipeline). Happy to draft it if you point me at the right one — otherwise it should go in the PR description as a merge blocker if you'd rather hold this until it exists.
One thing to re-check before you sign off: you verified the original file against the live site, and I've changed what it publishes, so a few of those verifications no longer apply to the new text. The directives are unchanged apart from an explicit Allow: /data-capture-sdk/ for the current tree — the rest is the file now being generated at build time instead of committed to static/.
| # | ||
| # GUIDES: every major stays open to every crawler, search engines included. | ||
| # People still run 6.x and 7.x and must find their documentation — in search, | ||
| # and through an assistant they ask. The guides ship exactly one tree per major |
There was a problem hiding this comment.
Contradicted by the file's own Sitemap block.
"The guides ship exactly one tree per major" vs. line 79, which documents 21 further live guide trees. Both cannot be true as written.
Confirmed live: https://docs.scandit.com/7.6.3/sdks/android/add-sdk/ and https://docs.scandit.com/6.28.10/sdks/android/add-sdk/ both return 200. (Probing the tree roots is misleading — /7.6.14/ itself 404s, since there is no directory index.)
Suggest "one CURRENT tree per major", or dropping the parenthetical.
|
Not sure if this is the right file, but is there a place we could make the agents browsing our docs + website aware that we have some published skills? |
Three review comments, plus two problems found while addressing them.
Generated by a plugin instead of committed to static/, which copies a file
verbatim. Two things a copy could not do:
- Track the versions. The Allow lines were hand-written, and the file's own
note said a major release "needs one line - a deliberate decision made
once". A decision made once is a decision forgotten once: retire 6.28 and
the Allow outlives the tree; freeze a new major and its readers lose their
API reference to the catch-all Disallow. Nothing would fail. They are now
derived from the docs versions in the build, through the same
linksToOwnApiLine the search tags use - so a version still linking the
unversioned tree does not get a line for a tree nothing points at.
- Tell a preview from production. Insurance, not a fix: previews are served
under a base_url subpath, so their robots.txt is inert today. Four lines to
cover a preview ever served at a host root.
Review comment 1: the published file advertised 21 retired guide trees.
Nothing links them and no sitemap carries them, so printing their addresses in
the most-fetched file on the host is a discovery list - the opposite of the
de-duplication the rest of the file argues for. Removed from the output, along
with the deploy postmortem, the search-index tag-rename history, the byte
counts and the build date. All of it now lives in the plugin source, which is
not served.
Review comment 2: inverted priority. Agreed, and it cannot be fixed here. The
~9,500 stale guide pages are the duplicate content that actually costs
something, and robots.txt is the wrong lever - blocking crawl on an indexed URL
strands it, unable to read the noindex that would resolve it. The fix is a
deploy that 410s them. The gap, its size and its owner are recorded in the
plugin source; it needs a ticket against the deploy pipeline.
Review comment 3: "the guides ship exactly one tree per major" contradicted the
census below it. Now "this build publishes one CURRENT tree per major", which
is accurate and derived. The contradiction also disappears with the census.
Found while doing it. The current tree had no Allow line: correct under RFC
9309, since `/*/data-capture-sdk/` needs a segment before the literal one and
the unversioned tree has none - but this group exists precisely for parsers
that publish no conformance statement, and the one failure that must not happen
is a sloppy `*` taking out the canonical tree while leaving the frozen ones
reachable. It is named explicitly now, for the same reason llms.txt is.
Also: deriving the sitemap counts by reading build/sitemap.xml was wrong. That
file is written by another plugin's postBuild, and plugin order put this one
first, so the read failed every build and the text degraded to "this build
emitted no sitemap" - a fallback that quietly said nothing. The counts are gone
from the published file anyway, per comment 1.
llms.txt and llms-full.txt are named in the file and allowed explicitly. The
naming is a comment: robots.txt has no registered field for them and nothing
discovers them from there - an agent finds /llms.txt by convention, the same
way it finds /robots.txt. The Allow lines are the functional part, so a future
blanket Disallow cannot remove them by accident.
Verified by building: production emits the rules above with no internals, a
preview emits Disallow: /, and the longest line is 80 characters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Agent Skills were already in llms.txt - ten entries, one per SDK - but only
nested inside each SDK's section, so nothing said "this site publishes Agent
Skills" until you had read the same line ten times in ten places. An agent
skimming for what Scandit offers had no top-level signal and no single URL to
fetch.
Two additions, both generated from the pages themselves:
- The blockquote at the top of llms.txt and llms-full.txt now says the skills
exist and where the index is. That blockquote is the one part of the
llmstxt.org layout an agent reads before the table of contents.
- /llms-agent-skills.txt, built from docs/sdks/**/agent-skills.mdx. An agent
asking "does Scandit ship skills for my host" should not have to pull
llms.txt (80 KB) or llms-full.txt (1.9 MB) and pick ten entries out of four
hundred. A new SDK appears here by having the page, not by anyone
remembering. Both are allowed explicitly in robots.txt.
scripts/test-robots.cjs asserts the generated output, which is what review
asked for when it said the measurements should be dropped or checked in
scripts/. Generating the file removed the hand-typed numbers; this removes the
rest of the assumption. It reads build/ and versions.json rather than
re-deriving from the config that produced the artifact - the same reason
test-search-facets.cjs reads the build - so it cannot agree with itself.
What it pins: the current tree is allowed explicitly; exactly the frozen
majors in versions.json get an Allow and no others, so a retired or newly
frozen major fails here instead of going quiet; the catch-all Disallow
survives; the llms indexes stay allowed; Allow precedes Disallow, which is
load-bearing for the first-match-wins parsers this group targets; search
engines stay unrestricted; and no internal narrative - retired trees, dates,
byte counts - leaks back into a file served on a public URL.
Each was mutation-checked: removing the current-tree Allow, removing a frozen
major's Allow, moving the Allow lines below the Disallow, and appending a
retired-tree census each turn the suite red.
Not addressed here, and not addressable here: the 21 retired guide trees. See
the note in docusaurus.config.ts and the PR discussion - that needs a deploy
that 410s them, not a rule in this file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Good question, and the answer is half yes — so let me split it, because the docs half is now done in this PR and the website half is not. robots.txt is not that place. It has no registered field for this, and crawlers ignore unknown directives, so anything we wrote there would look like it worked without doing anything. robots.txt controls permission (may you fetch this path), never advertising (here is what we publish). llms.txt is that place, and the skills were already in it — just not findably. All ten Agent Skills pages were listed, but only nested inside each SDK's own section. So nothing said "this site publishes Agent Skills" until you had read the same line ten times in ten places, and there was no single URL to fetch. Two changes in this PR:
Both are named explicitly in robots.txt so a future blanket How an agent actually finds these: by convention, not by us pointing. The website is not covered, and I can't fix it from this repo.
Happy to write that up as a ticket if you tell me who owns it. |
PR 431 merged while this branch was open, so both had appended to the scripts block in package.json and to the test steps in build-docs.yml. Both conflicts are two additions to the same list and neither side loses anything. The CI steps are ordered cheapest-first: test:robots reads two files already on disk, while the API-reference suites spawn child processes and stub origins. Review findings on the test, all three of them mine: The expectation disagreed with the rule it was checking. It derived the allowed API lines from versions.json while robotsTxtPlugin derives them from linksToOwnApiLine - a content scan that deliberately skips a frozen version whose snapshot still links the UNVERSIONED tree. Those diverge in the normal release shape: 8.6.0-beta.1 added 8.5.2 to versions.json, and until the freeze rewrites that snapshot's links it correctly gets no Allow line, because it sends readers somewhere already open. The test would have demanded one and failed the build on main, blaming the generated file for being right. It now reads the snapshots in versioned_docs/ directly - the same ground truth, a different implementation, so it is a second opinion rather than the same derivation twice. Verified both ways against a simulated freeze: a snapshot still linking the unversioned tree expects no Allow, and one linking its own line demands it. The table-of-contents guard could not fire. `indexOf` returns -1 when the heading is missing and `slice(0, -1)` is then the whole document, so the check silently widened from the blockquote to all 400 entries - where "agent skills" appears anyway, making it pass while the thing it protects was gone. The heading is a hardcoded literal in docusaurus-plugin-llms, so an upgrade can move it. Guarded before slicing. The per-SDK count ignored llmsIgnoreFiles. docs/sdks/titanium is on that list, so adding an agent-skills page there would have failed CI claiming the index was wrong when it was right. The ignore globs are now read out of the config - scoped to the ignore declarations, because scraping the whole file swept up this index's own includePatterns, whose literal prefix is "docs/sdks/" and which then matched every page and made the expectation zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every link this PR publishes was a 404.
docusaurus-plugin-llms hardcodes a "docs" segment into every URL it emits, and
this site sets routeBasePath: "/" on the docs plugin, so pages are served at
/sdks/... and not /docs/sdks/... . Checked against the live site:
/docs/sdks/android/agent-skills is 404, /sdks/android/agent-skills is 200.
llms.txt and llms-full.txt have carried that prefix since they were first
generated - roughly 400 dead links each - and it was survivable while nothing
pointed at them. This PR is what made it matter: llms-agent-skills.txt is a
file whose entire payload is ten of those links, announced in the blockquote
and allowed by name in robots.txt, so an agent following the new pointer would
have found ten 404s. Exactly the opposite of the point.
`pathTransformation: { ignorePaths: ["docs"] }` repairs all three. Verified
after the build: zero remaining /docs/ prefixes across the three files, and all
ten agent-skills URLs return 200 live.
crawlableApiLines no longer skips the served version. The skip read "served
unversioned", but whether a version's pages link the unversioned tree is
exactly what linksToOwnApiLine already decides from its content - redundant
where the two agreed and wrong where they did not.
scripts/update-version.py rewrites DOCS_LAST_VERSION from "current" to a
numbered version during a beta window, so the served version becomes a real
entry here and a directory in versioned_docs/. If the freeze has rewritten that
snapshot's links to its own line, the root-served guides point at
/8.5/data-capture-sdk/ while the catch-all Disallow blocks it and the Allow
covers a tree nothing links - backwards, and silent. The test's ground truth
grew to match: it scans the current docs/ tree too, taking the served number
from build/search-tags.json rather than from the config that wrote robots.txt.
And the ignore-block scrape in the test is now anchored on its own terminator.
`indexOf("\n]")` looked scoped and was not - llmsIgnoredSdkTrees is written on
one line, so its block ran forward into whatever array came next, and declaring
another one in between would have changed the answer silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five findings, and the two that matter are both in the function added last round to fix last round's finding. docs/ was paired with the wrong version number. It is ALWAYS the `current` version, but the test looked it up through `lastVersionTag`, which follows DOCS_LAST_VERSION - and update-version.py rewrites that from "current" to a number during a beta window, which is the window this check exists for. In that state the tag names the frozen release while docs/ still holds the unreleased one, so the test scanned the beta tree for the previous release's line and disagreed with the generator. Demonstrated: with lastVersionTag "docs-default-8.6.0" the old lookup pairs docs/ with 8.6.0 while docs/ holds 8.7.0. It now reads `docs-default-current`, which is what docs/ is by definition. The two assertions contradicted each other. The equality deliberately lets the current tree contribute an Allow line, and the guard below it was built from versions.json, which never contains the current version - so the moment the current tree did contribute, deepStrictEqual passed and the guard hard-failed on the same line. Dropped: the equality is already exact, and every line in the expectation came from a tree on disk. The test enumerated versioned_docs/ while the generator enumerates docsVersions, so a snapshot directory left on disk but dropped from versions.json - the config records version-8.5.3 in exactly that state - made the test demand an Allow the generator will never emit. It now takes the frozen trees from versions.json, which also removes an ENOTDIR crash on a stray file in that directory. Verified: a stale version-8.5.3 no longer votes. Two comments were wrong rather than the code. The pathTransformation note described stripping a prefix, but the plugin applies it to the whole doc-relative path, so a future route containing a segment named "docs" would lose it too - latent today, and now written down where it would break. And the llms-agent-skills.txt existence check now says why it is not a formality: generateCustomLLMFiles only warns and writes nothing when its patterns match no docs, and the plugin swallows that, so a rename would ship a robots.txt and an llms.txt both pointing at a 404 with a green build. That assertion is the only thing that catches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… guard
Two findings, both the same shape: a guard whose comment claimed more than the
code delivered.
The ignore-pattern scrape read two of the four lists. llmsIgnoreFiles assembles
llmsLinuxPartialIgnore and the llmsNonWebSdkRoots x llmsSharedPartialPageNames
cross-product by spreading identifiers rather than literals, so their patterns
were never seen - while the comment above it said "adding one does not silently
loosen this", which was true only for the titanium list. The regex also
required a trailing `*`, so a non-glob entry would have been invisible for the
same reason.
Rather than patch the scrape a third time, it is gone. llms.txt is produced by
the same plugin from the same ignore set, so any agent-skills page it lists is
one that was not ignored; comparing the two build outputs needs no knowledge of
how the ignore lists are written and cannot drift when someone adds a fifth.
The check also confirms every indexed URL maps to a page this repo builds, so
the index cannot advertise something nothing produces. Verified: dropping a
page from the index while it remains in llms.txt fails the suite.
The banned-number guard was /\b\d{2},\d{3}\b/, which caught the byte counts and
none of the other measurements its own comment named as the thing to keep out -
the sitemap census was 1,508 / 618 / 498 and the stale-tree figure ~9,500.
Someone re-adding "the sitemap carries 1,508 guide URLs" at a release would
have passed and shipped the number to the most-fetched URL on the host. Now any
comma-grouped number, plus a bare count with a unit. Deliberately not any long
number: RFC 9309 is cited in the file and is not a measurement. Verified all
three previously-missed figures are caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…them
VERIFICATION STATUS, stated up front: the glob change is verified directly
against minimatch; the three test changes are NOT yet exercised against a
build. The build that would have confirmed them was killed by the system for
low memory, which cleared build/, and it has not been re-run. Treat the test
changes as reviewed but unproven until `yarn build && yarn test:robots` has
run.
The index glob matched only .mdx. All ten skills pages are .mdx today so
nothing is broken now, but docusaurus-plugin-llms indexes .md and .mdx alike
and .md is the dominant convention in this tree - 448 files against 123. An SDK
adding docs/sdks/<new>/agent-skills.md would therefore land in llms.txt and not
in llms-agent-skills.txt: the file robots.txt advertises would quietly omit that
SDK, and the gate would fail the build blaming the index rather than the
pattern. Now `agent-skills.{md,mdx}`, checked against the installed minimatch
for both extensions and the two-level sdks/net/* paths.
Three guards were narrower than their own stated purpose:
The existence probe hardcoded `.mdx`, which was unreachable only because the
include pattern was .mdx-only as well - widening one without the other would
have failed a correct, building page. It now probes .md, .mdx and the index.*
forms.
The banned-narrative guard named /7.6.x/ and /6.28.x/ literally, and those are
exactly the lines that stop being the retired ones. After the next major
transition a discovery list for the then-frozen major would have passed every
pattern in the check written to stop precisely that. Derived now: any
three-part tree path. Two-part lines like /8.7/ stay legal, because the file
writes them when explaining the rule.
And llms.txt was read without the existence guard its two siblings get, so the
failure this suite exists to diagnose - the llms plugin not producing its files,
which it logs rather than failing on - arrived as a raw ENOENT stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The commit before this said its three test changes were reviewed but not
exercised: the build that would have confirmed them was killed by the system
for low memory, which cleared build/. The build has now been re-run at the
user's request and the changes are confirmed, so the caveat no longer stands.
Build clean. Suites: test:robots 10, test:search-facets 10,
test:api-reference-seo 44, test:api-reference-seo-verdicts 15.
Each change exercised rather than inferred from a green run:
- The derived banned-pattern catches a discovery list for a major that is not
retired yet - "/8.6.1/ through /8.6.9/" - which the previous hardcoded
/7.6.x/ and /6.28.x/ list would have passed straight through. That is the
regression the check exists to stop, one major transition from now.
- Removing build/llms.txt now fails with "build/llms.txt is missing -
docusaurus-plugin-llms did not produce it", not a raw ENOENT stack.
- The widened existence probe still passes against the real index, so
accepting .md and the index.* forms did not loosen it into a no-op.
Still verified by construction rather than end to end: a new SDK authoring
agent-skills.md. minimatch matches the {md,mdx} glob for both extensions and
the plugin indexes both (utils.js:98), but no build has been run with such a
page present - that would have cost a second build while the machine was short
of memory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 participants
/robots.txt returns 404 today. A missing robots.txt is not neutral — crawlers read it as "everything permitted" — so the site's policy was in force by omission rather than stated. This writes it down and declares the sitemap.
Guides are untouched: every major stays open to every crawler. People still run 6.x and 7.x and must find their documentation, in search and through an assistant they ask. The guides already ship one tree per major (/ for 8.x, /7.6.14/, /6.28.11/).
The API reference is a separate site on the same host — Sphinx, not Docusaurus, built and deployed by another pipeline. robots.txt is per-host, so this one file governs both. Unlike the guides it ships a tree per MINOR version: six versioned trees are live alongside the unversioned one. Each was verified by fetching ONE real page from it (index.html is only a redirect stub, so probing it proves nothing): that page is byte-identical at 38,575 bytes between the unversioned tree and /8.6/, is ~38 KB in each of the frozen 8.x minors /8.5/ /8.4/ /8.3/, and ~35 / ~32 KB in /7.6/ and /6.28/. Those are one-page measurements — no claim is made about tree completeness. The current version always lives at the unversioned path, so /8.6/ is a pure duplicate.
The rule is default-deny and self-maintaining rather than a version list that goes stale:
A new minor is excluded with no edit, and its content is already served at the unversioned path. A new major is also excluded with no edit — the Disallow's
*matches whatever the first path segment is, so /9.0/, /9.1/, … are each caught individually as they appear; there is no "/9.x/" pattern involved. Keeping the outgoing major's tree then costs one Allow line, decided once at the release.The Allow lines are placed FIRST deliberately. Under RFC 9309 longest-match precedence the order does not matter, because the longer Allow beats the shorter Disallow either way. But this group deliberately excludes Googlebot and Bingbot, and none of the six agents in it publishes a conformance statement, so no RFC 9309 guarantee applies to them. A first-match-wins parser (historical Nutch / crawler-commons behaviour, which is CCBot's lineage) would read a leading Disallow and deny /7.6/ and /6.28/ outright, silently breaking the requirement that every major stays reachable. Evaluated under both models, the six cases that matter now agree:
With the Disallow first, first-match-wins denied /7.6/ and /6.28/ — the only two rows that changed.
Bulk crawlers only, and the usual objection does not apply here. Blocking crawl on an indexed URL can strand it, so the standard advice is to use canonical/noindex instead — but the Sphinx pages carry no and no canonical link at all, and adding them means changing the Sphinx build, not this repository. robots.txt is the only lever reachable from here. It is applied to bulk crawlers, where stranding is irrelevant because nothing is ranking them; search-engine de-duplication is left to the Sphinx side rather than half-done here.
No Crawl-delay. No requirement asks for rate limiting, only two of the six named agents document support for it, and Amazonbot's own documentation says it does not support the directive — so the directive would have thrown an inaccurate claim into the file while throttling ClaudeBot and CCBot across the whole site, including the guides the "reachable by assistants" requirement wants open.
The file states no page count for the API reference. The "~3,200" and "~3,900" figures elsewhere in this repo both count pages that dropped out of the SEARCH INDEX during a tag rename; neither is a census of sitemap-absent pages. What is recorded instead is measured: the sitemap carries 1,508 guide URLs (618 root, 498 /7.6.14/, 392 /6.28.11/) and zero /data-capture-sdk/ URLs, so the API reference is in no sitemap and is link-discoverable only, with three trees link-reachable. Also noted: 21 retired version trees are still live and unsitemapped — /7.6.3/–/7.6.13/ and /6.28.1/–/6.28.10/, roughly 450-500 pages each — while /8.5.3/, /8.5.2/, /8.4.1/ and /8.3.1/ return 404.
Docusaurus copies static/ to the build root — verified live, where static/img/logo-light.svg serves as /img/logo-light.svg. Deployment and cache invalidation happen outside this repository, and a second pipeline writes the Sphinx trees to the same host, so confirm https://docs.scandit.com/robots.txt returns 200 after deploy and is not later clobbered by that pipeline.