Skip to content

test: use a non-credential-shaped sentinel in the leak tests - #2

Merged
dmccoystephenson merged 1 commit into
mainfrom
fix/leak-test-fixture-not-credential-shaped
Sep 1, 2026
Merged

test: use a non-credential-shaped sentinel in the leak tests#2
dmccoystephenson merged 1 commit into
mainfrom
fix/leak-test-fixture-not-credential-shaped

Conversation

@dmccoystephenson

Copy link
Copy Markdown
Contributor

Why

js/test/client.test.ts and python/tests/test_client.py both hand the client a stand-in secret and assert it never reaches a returned value, a request URL, or an error message. Good tests — but the stand-in was a literal ghp_-prefixed string.

That was harmless while this repo was private. Now that it is public, GitHub secret scanning matches the literal, files a false-positive alert, and push protection can block any future push that touches those lines.

The alert is noise. The problem with recurring noise is that it trains people to dismiss the alert that is real.

What changed

The fixture value only, in both suites:

- 'ghp_a1b2c3d4e5f6g7h8i9j0_secretvalue'
+ 'sentinel-not-a-real-credential-leak-canary'

Both files gained a comment explaining that the value deliberately avoids the ghp_ / gho_ / github_pat_ prefixes and any random-looking entropy, so nobody later "fixes" it back into something authentic-looking.

What did not change

The assertions. The tests never needed the fixture to look like a credential — they needed a distinctive, greppable sentinel to assert the absence of, and the replacement is exactly as good at that. Every call site (Authorization: Bearer ${TOKEN}, the leaky-error-body construction, the URL check, the not.toContain / assertNotIn assertions) is untouched.

Verification

Both suites pass, and the leak assertion was mutation-checked: with GitHubDocsClient._redact stubbed out to return its input unchanged, test_an_upstream_error_that_echoes_the_token_is_redacted fails on the new sentinel exactly as it would have on the old value. The redaction was restored before committing.

$ cd js && npx vitest run
 ✓ test/catalogue.test.ts  (12 tests) 16ms
 ✓ test/markdownUrl.test.ts  (10 tests) 55ms
 ✓ test/urls.test.ts  (19 tests) 29ms
 ✓ test/client.test.ts  (24 tests) 63ms

 Test Files  4 passed (4)
      Tests  65 passed (65)
$ cd python && python3 -m unittest discover -s tests -v
test_an_upstream_error_that_echoes_the_token_is_redacted (test_client.TestCredentialHandling) ... ok
test_token_travels_in_a_header_and_never_in_a_url (test_client.TestCredentialHandling) ... ok
...
----------------------------------------------------------------------
Ran 31 tests in 0.118s

OK

The leak tests hand the client a stand-in secret and assert it never
reaches a returned value, a URL or an error message. The stand-in was
literally `ghp_...`-prefixed. That was fine while the repo was private;
now that it is public, GitHub secret scanning matches the literal, files
a false-positive alert, and push protection can block any push that
touches the line.

The alert is noise, but recurring noise is worse than no alert: it
trains people to dismiss the one that is real.

The tests are unchanged in what they check. They only ever needed a
distinctive, greppable sentinel to assert the absence of, and
`sentinel-not-a-real-credential-leak-canary` does that job identically
while matching no scanner pattern and carrying no entropy. Both files
carry a comment explaining why the fixture deliberately does not look
authentic, so it does not get "corrected" back later.

Verified by removing the redaction in the Python client: the leak test
fails on the sentinel exactly as it did on the old value.
@dmccoystephenson
dmccoystephenson merged commit 8ce068e into main Sep 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant