watcher

Long-running asset monitoring toolkit for DNS records, TCP ports, web services, web paths, URL status, and lightweight exposure checks.

中文文档: README.zh-CN.md

This PROJECT was written with assistance from Codex AI.

Overview

watcher is a conservative security operations CLI for keeping a baseline of business-facing assets and detecting changes over time. It stores imported and discovered assets in a local SQLite database, runs scheduled monitoring batches, generates human-readable reports, and can send the report package by email.

It is designed for teams that need to answer questions like:

• Did any domain start resolving to a different IP?
• Did a new TCP port open on a known asset?
• Which open ports look like web services?
• Did directory enumeration discover non-baseline URLs?
• Did a lightweight POC find an exposed sourcemap?
• What changed since the imported asset baseline?

watcher favors slow, low-noise probing over aggressive scanning. Use it only on assets that you own or are authorized to assess.

Features

• Baseline asset inventory: import domains, real IPs, bound IPs, ports, and URLs from Excel, or manage individual baseline assets from the CLI.
• Business-system centric model: assets are grouped by system so reports remain useful for ownership and remediation.
• DNS monitoring: resolves configured domain names and records DNS change alerts.
• Slow TCP port monitoring: scans configured ports with per-IP and per-port concurrency controls.
• Service fingerprinting: identifies HTTP/HTTPS services and captures simple banner/fingerprint details.
• Detailed service fingerprinting: optional nmap -sV service detection runs after the lightweight fingerprint phase.
• Web path enumeration: uses a path dictionary, filters common fake-200 responses, and can discover links from HTML and JavaScript.
• Lightweight vulnerability checks: currently includes webpack_sourcemap_disclosure.
• SQLite persistence: local, portable storage with built-in migrations.
• Reports: always generates summary.md, plus detail files in xlsx, json, or csv, then packages everything as a zip.
• SMTP notification: sends batch summaries and report zip attachments when enabled.
• MCP inventory server: expose confirmed-live ports, web services, URL status, alerts, and findings to LLM hosts over stdio. The server is read-only and does not scan or exploit targets.
• Static-build friendly dependencies: uses rustls for HTTP/SMTP TLS and bundled SQLite through rusqlite.

Status

The project is currently an early 0.1.0 implementation. The core workflow is usable, but command output, report structure, and POC coverage may still evolve.

Installation

Requirements

• Rust toolchain with Cargo
• A platform supported by tokio, rusqlite, and reqwest

Build from source

git clone <repository-url>
cd watcher
cargo build --release

The binary will be available at:

target/release/watcher

For local development, you can also run commands through Cargo:

cargo run -- --help

Release build script

scripts/build_release.sh resolves paths from the repository root and keeps release artifacts in target/ by default. Build for the current host while developing:

scripts/build_release.sh --target host

To build the supported Linux musl and Windows GNU artifacts, omit --target or select them explicitly. The script checks Rust targets and reports a missing cross-linker before Cargo builds. Use --dry-run to inspect the commands without requiring cross-compilation toolchains.

scripts/build_release.sh --target linux --target windows
scripts/build_release.sh --dry-run --target linux

Quick Start

Create the default config and database:

watcher init

Print an example YAML config:

watcher --example

Import baseline assets from Excel:

watcher import --type excel ./assets.xlsx

Import a web path dictionary:

watcher import --type path ./paths.txt

Run one monitoring batch:

watcher task run --once

Query recent logs:

watcher query --type log --limit 50

Build a report package for the latest batch:

watcher report

Default Paths

watcher uses the user config directory by default:

• Config: ~/.config/watcher/watcher.yml
• Database: ~/.config/watcher/watcher.db
• Reports: ~/.config/watcher/reports
• Daemon PID file: next to the config file as watcher.pid

Running watcher init or any normal command creates missing config/database paths automatically.

Asset Model

The database is organized around business systems:

• systems: business systems and ownership boundaries.
• domains: domain names and their expected/latest resolved IPs.
• ip_addresses: real IP addresses and resolved IP addresses.
• ports: only imported/baseline and currently open TCP ports, plus service protocol, web flag, and fingerprint. Full-port scans do not persist closed unknown ports.
• urls: imported URLs, enumerated URLs, JS-discovered URLs, and vuln URLs.
• dict_paths: web enumeration dictionary entries.
• batches: monitoring batch status and report package path.
• alerts: DNS, port, and vulnerability events. Port changes are stored once per IP with the changed port list in details, not once per port.
• scan_summaries: one compact row per IP per batch (probed_ports, open_count, opened/closed port lists) instead of per-port scan logs.
• vulnerabilities: lightweight POC findings.
• pending_work: carry-over work when a batch stops before completion.
• logs: application logs stored in SQLite for CLI query/export.

domains, ip_addresses, ports, and urls include an is_baseline marker. Imported assets are baseline assets; assets found by scanning, enumeration, or POCs are non-baseline discoveries by default.

Excel Import

The first worksheet should contain these headers:

id,system,servername,real_ip,servername_bind_ip,port,url

Column behavior:

• id: ignored.
• system: business system name. Required.
• servername: domain name. Optional.
• real_ip: real asset IP address.
• servername_bind_ip: expected or previously known domain binding IP.
• port: TCP ports. Supports separators such as 80,443/8080.
• url: related URL. Optional.

Example:

watcher import --type excel ./assets.xlsx

Asset Management

Asset, dictionary, and log operations use action-first commands. Select the noun with --type / -t. Mark baseline assets with --baseline.

watcher add --type url --baseline --system core https://example.com/login
watcher import --type url --baseline --system core ./urls.txt
watcher query --type ip --baseline --keyword 10.0.0

watcher add --type ip --baseline --system core 10.0.0.1
watcher add --type port --baseline --system core --ip 10.0.0.1 443
watcher import --type port --baseline --system core --ip 10.0.0.1 ./ports.txt
watcher add --type name --baseline --system core --bind-ip 10.0.0.1 example.com

watcher unmark --type port --system core --ip 10.0.0.1 8443
watcher delete --type url --system core https://example.com/old

delete --type url|port|ip|name --system <system> removes the asset row. unmark keeps the row but changes it from baseline to non-baseline.

Add a dynamically discovered non-baseline asset by omitting --baseline:

watcher add --type url --system core https://example.com/admin
watcher add --type port --system core --ip 10.0.0.1 8443

Business Systems

watcher add --type system core
watcher query --type system --keyword core
watcher export --type system ./systems.csv
watcher rename --type system core core-prod
watcher delete --type system core-prod

query --type system and export --type system include domain, IP, port, URL, and baseline counters. Deleting a system cascades to assets under that system.

Monitoring Pipeline

Each batch runs the following stages:

1. DNS resolution: resolves every domain and records DNS changes.
2. Port scanning: probes configured TCP ports on real IP assets and records open/closed changes.
3. Fingerprinting: checks open ports for HTTP/HTTPS and banner details.
4. Web enumeration: tries dictionary paths against web services, including ip:port and same-system name:port forms, then extracts possible links from HTML/JS.
5. Lightweight POCs: runs exposure checks in parallel with web enumeration.
6. Report packaging: writes summary.md and detail files, then creates a zip package.
7. Email notification: sends the summary and zip attachment when enabled.

Configuration

Use watcher --example to print the default YAML config, or see examples/watcher.yml.

Display Timezone

Watcher stores timestamps in UTC and renders human-facing output with display.timezone. The default is UTC+08:00:

display:
  timezone: "+08:00"

Accepted values include fixed offsets such as +08:00, -05:30, UTC+8, and UTC.

Probe Settings

Port scanning uses two concurrency controls:

probe:
  connect_timeout_ms: 2000
  scan_ip_concurrency: 4
  scan_port_concurrency_per_ip: 4

Effective port probe parallelism is:

scan_ip_concurrency * scan_port_concurrency_per_ip

probe.concurrency is still used by non-port tasks such as fingerprinting, web enumeration, and vulnerability checks.

Detailed fingerprinting is disabled by default because it depends on nmap. When enabled, it starts after the lightweight fingerprint phase and runs in parallel with web enumeration and vulnerability checks:

fingerprint:
  detailed:
    enabled: true
    nmap_path: nmap
    timeout_ms: 30000
    concurrency: 2

Configure explicit ports:

probe:
  scan_ports:
    - 80
    - 443
    - 8080

Or scan all TCP ports:

probe:
  scan_ports: full

full and all both expand to 1..=65535. Full-port scans can take a long time; use longer scheduler intervals and lower concurrency for conservative monitoring.

DNS Servers

An empty dns_servers list uses the host/system resolver:

probe:
  dns_servers: []

Custom DNS servers support IP and IP:port forms:

probe:
  dns_servers:
    - 8.8.8.8
    - 1.1.1.1:53

Reports

summary.md is always generated. Details are controlled by report.format:

report:
  output_dir: ~/.config/watcher/reports
  format: xlsx

Supported formats:

• xlsx: one details.xlsx workbook with alerts, vulnerabilities, urls, and open_ports sheets.
• json: one structured details.json file.
• csv: separate detail CSV files.

POC Switches

Each lightweight vulnerability POC can be enabled or disabled independently. Currently the built-in POC list contains webpack_sourcemap_disclosure:

pocs:
  webpack_sourcemap_disclosure:
    enabled: true
    max_urls_per_batch: 1000
    max_js_files_per_url: 20
    max_map_candidates_per_url: 20

Set enabled: false to skip that POC without affecting other monitoring tasks. The limit fields bound task5 work for large URL histories or pages with many JavaScript files.

Daemon and Tasks

Run continuously in the background:

watcher daemon run

Run in the foreground for debugging:

watcher daemon run --foreground

Manage daemon state:

watcher daemon status
watcher daemon stop
watcher daemon restart

Manage task batches:

watcher task run --once
watcher task list
watcher task status
watcher task stop

Interactive Dashboard

Open the live terminal dashboard in an interactive shell:

watcher dashboard
watcher dashboard --refresh-seconds 5

The dashboard refreshes asset inventory, open-port and web-service counts, latest batch status, per-stage task progress, pending-work queue state, alert severity totals, vulnerability counts, and recent alerts. Press q or Esc to exit. Critical, high, medium, and low findings use distinct colors.

MCP for LLM Hosts

watcher can serve its local asset inventory over the Model Context Protocol. Use this when a large-language model should plan authorized testing against assets that watcher has already marked live.

watcher mcp

stdio is reserved for JSON-RPC. Logs go to stderr and SQLite. The server is read-only: it does not start scans, send probes, or run exploits.

Natural-language examples for LLM hosts: docs/mcp.example.md.

Example Claude Desktop / Cursor config:

{
  "mcpServers": {
    "watcher": {
      "command": "watcher",
      "args": ["mcp"]
    }
  }
}

Useful tools. All asset lists are paginated (limit default 50, max 200) and return items, total, has_more, and next_offset. Follow next_offset instead of raising limit when more rows exist.

• get_live_inventory: one page of open TCP ports, HTTP(S) services, and 2xx/3xx URLs
• get_system_context: one page of a business system's live assets and findings
• list_live_ports / list_web_services / list_live_urls: continue a specific live/web list
• list_alerts / list_vulnerabilities: latest monitoring findings

Prompts pentest_live_assets and review_web_exposure inject the first page into an authorized-testing brief. Only assess assets you own or are authorized to test.

Logs

Runtime logs are stored in SQLite and can be queried or exported:

watcher query --type log --level info --limit 100
watcher query --type log --level error --keyword smtp --limit 20
watcher export --type log ./watcher-logs.csv --limit 1000
watcher clear --type log --before 2026-05-15T00:00:00Z

Email Notification

SMTP notifications are optional. QQ/Foxmail users should use an SMTP authorization code instead of the web login password.

smtp_security: auto maps port 465 to implicit TLS/SMTPS and port 587 to STARTTLS:

email:
  enabled: true
  smtp_host: smtp.qq.com
  smtp_port: 465
  smtp_security: auto
  username: your-account@qq.com
  password: "your-smtp-authorization-code"
  from: your-account@qq.com
  to:
    - security@example.com

For mail troubleshooting:

watcher query --type log --keyword email --level warn --limit 20
watcher query --type log --keyword smtp --limit 20

Command Reference

watcher init
watcher --example

watcher import --type excel <file>
watcher add --type url|port|ip|name --system <system> [--baseline] <value>
watcher import --type url|port|ip|name --system <system> [--baseline] <file>
watcher query --type url|port|ip|name [--baseline]
watcher export --type url|port|ip|name [--baseline] <file>
watcher delete --type url|port|ip|name [--system <system>] <value>
watcher unmark --type url|port|ip|name --system <system> <value>

watcher add --type system <name>
watcher query --type system
watcher export --type system <file>
watcher rename --type system <old> <new>
watcher delete --type system <name>

watcher daemon run|status|stop|restart
watcher task run|list|status|stop
watcher mcp
watcher dashboard


watcher query --type log [--level] [--keyword] [--limit]
watcher export --type log <file>
watcher clear --type log [--before]

watcher import --type path <file>
watcher export --type path <file>
watcher query --type path
watcher delete --type path <path>

watcher report

Use add for dynamically discovered individual assets. These assets remain non-baseline unless --baseline is set.

Use watcher <command> --help for exact arguments.

Development

Run checks:

cargo fmt --check
cargo clippy -- -D warnings
cargo test

Run the CLI locally:

cargo run -- --help
cargo run -- --example
cargo run -- task run --once

Roadmap

• Broader POC registry and clearer extension points.
• Integration tests with mock HTTP services and sample Excel files.
• Systemd service examples for daemon deployment.
• More granular per-stage task progress.
• Richer report comparison sections for newly discovered assets.

License

MIT. See LICENSE.