Skip to content
BranchesTags

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
offline-demo
offline-demo
 
 
src
src
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
package-lock.json
package-lock.json
 
 
package.json
package.json
 
 

Repository files navigation

Spiderweb

Spiderweb is a small interactive web scraper CLI built with Node.js and Playwright. It keeps a browser page open for the whole session, so you can navigate a site, inspect its HTML, click elements, and capture fragments from the current DOM.

Setup

  1. Install the package dependency:

    npm install playwright

  2. Install the Chromium browser binary Playwright uses:

    npx playwright install chromium --no-shell

    This project launches Playwright with Chromium's newer headless mode, so you do not need the separate chromium-headless-shell download.

  3. Start the CLI:

    npm start

You can also open an initial URL right away:

npm start -- example.com

Or run a visible browser window while you experiment:

npm start -- --headed

You can also try the included offline demo page:

npm start -- [path]/spiderweb/offline-demo/demo.html

Commands

help
navigate <url>
goto <url>
show code
capture <css-selector>
click on <css-selector>
text <css-selector>
attr <css-selector> <attribute-name>
images [limit]
links [limit]
title
state
whereami
wait for <css-selector>
screenshot [file-path]
back
reload
history
export json [file-path]
export all [file-path]
download images [directory]
quit
exit

Example sessions

The examples below cover every command. Real-site outputs were captured from nodejs.org on April 4, 2026, so titles and page markup may change as the docs are updated.

Session 1: real website

$ npm start
Spiderweb is ready.
Browser mode: headless
Type "help" to see the available commands.

scraper> navigate https://nodejs.org/api/readline.html
Opened: https://nodejs.org/api/readline.html
Status: 200
Title: Readline | Node.js v25.9.0 Documentation

scraper> title
Readline | Node.js v25.9.0 Documentation

scraper> state
URL: https://nodejs.org/api/readline.html
Title: Readline | Node.js v25.9.0 Documentation

scraper> whereami
URL: https://nodejs.org/api/readline.html
Title: Readline | Node.js v25.9.0 Documentation

scraper> capture h1
Matches: 1
<h1>Node.js v25.9.0 documentation</h1>

scraper> text h1
Matches: 1
Node.js v25.9.0 documentation

scraper> attr link[rel="canonical"] href
Matches: 1
https://nodejs.org/api/readline.html

scraper> links 3
1. Skip to content
   https://nodejs.org/api/readline.html#apicontent
2. Node.js
   https://nodejs.org/
3. About this documentation
   https://nodejs.org/api/documentation.html

scraper> wait for h1
Element appeared: h1

scraper> goto https://nodejs.org/api/path.html
Opened: https://nodejs.org/api/path.html
Status: 200
Title: Path | Node.js v25.9.0 Documentation

scraper> reload
Reloaded: https://nodejs.org/api/path.html
Status: 200

scraper> exit

Session 2: local demo page

$ npm start -- [path]/spiderweb/offline-demo/demo.html
Spiderweb is ready.
Browser mode: headless
Type "help" to see the available commands.
Opened: [path]/spiderweb/offline-demo/demo.html
Title: Spiderweb Demo

scraper> help
Spiderweb commands

  help
    Show this help menu.

  navigate <url>
  goto <url>
    Open a page. If you omit the protocol, https:// is assumed.

  ...

  images [limit]
    List page images, including CSS background images.

  ...

  export json [file-path]
    Save the last successful command result as JSON. Defaults to exports/.

  export all [file-path]
    Save all successful command results from this session as JSON. Defaults to exports/.

  download images [directory]
    Download all discovered page images. Defaults to downloads/.

  quit
  exit
    Close the browser and leave the CLI.

scraper> show code
<!DOCTYPE html><html lang="en"><head>
    <meta charset="utf-8">
    <title>Spiderweb Demo</title>
    <style>
      ...
    </style>
  </head>
  <body>
    <main>
      <h1>Spiderweb Demo</h1>
      <p class="summary">This page exists so the CLI can be tested offline.</p>
      <section class="gallery">
        <img src="./assets/spider.svg" alt="Spider illustration">
        <img src="./assets/web.svg" alt="Web illustration">
      </section>
      <div class="hero-panel" aria-label="Panel background"></div>
      <a class="more-link" href="./details.html">Open details</a>
      <button class="action-button is-hidden" type="button">
        Hidden action
      </button>
      <button id="hello-button" class="action-button" type="button" onclick="document.querySelector('.summary').textContent = 'The visible action button changed this text.';">
        Change summary
      </button>
    </main>
  ...

scraper> images
1. [img] Spider illustration
   file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/spider.svg
   200x150
2. [img] Web illustration
   file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/web.svg
   200x150
3. [background] Panel background
   file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/panel.svg
   280x140

scraper> click on .action-button
Clicked first visible match out of 2 total matches.
Visible matches: 1

scraper> text .summary
Matches: 1
The visible action button changed this text.

scraper> click on a.more-link
Clicked first visible match out of 1 total matches.
Visible matches: 1
Current URL: file:///Users/daniel/Desktop/spiderweb/offline-demo/details.html

scraper> back
Navigated back to: file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html
Title: Spiderweb Demo

scraper> history
1. about:blank
2. file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html
3. file:///Users/daniel/Desktop/spiderweb/offline-demo/details.html
4. file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html

scraper> screenshot
Saved screenshot to /Users/daniel/Desktop/spiderweb/screenshots/screenshot-2026-04-04T18-59-11-123Z.png

scraper> export json
Saved JSON export to /Users/daniel/Desktop/spiderweb/exports/result-2026-04-04T18-59-14-456Z.json

scraper> export all
Saved full session export to /Users/daniel/Desktop/spiderweb/exports/session-2026-04-04T18-59-16-789Z.json

scraper> download images
Downloaded 3 of 3 images to /Users/daniel/Desktop/spiderweb/downloads/images-2026-04-04T19-31-54-748Z

scraper> quit

JSON export

export json [file-path] writes only the last successful command result to disk. That means you can:

  • run links 20, then export the link list as JSON
  • run text .headline, then export the extracted text
  • run attr a.cta href, then export the selected attribute with page metadata

Each single-result export includes:

  • the command you ran
  • the current page URL and title
  • the scraped result
  • an exportedAt timestamp

If you omit the file path, Spiderweb writes the file into exports/ automatically with a timestamped filename.

export all [file-path] writes the full session result history so far. It includes:

  • sessionStartedAt
  • exportedAt
  • entryCount
  • entries, where each entry contains recordedAt, command, page, and result

If you omit the file path, Spiderweb writes the file into exports/ automatically with a timestamped filename.

Screenshots

screenshot [file-path] saves a full-page PNG. If you omit the file path, Spiderweb writes the image into screenshots/ automatically with a timestamped filename.

Images

images [limit] lists images on the current page. It includes normal <img> elements and CSS background images, along with a label, source URL, and dimensions.

download images [directory] downloads all discovered page images, including CSS background images. If you omit the directory, Spiderweb writes them into downloads/images-<timestamp>/.

About

Web Crawler CLI

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages