Bot Or Not? Detecting AI-Driven Web Traffic Using Network Metadata Analysis

This repository contains files and information related to the SANS Technology Institute research project for identifying human or bot internet traffic with only high-level network metadata.

The folders contain all the settings and scripts used in the course of the research:

web_server: All files and configurations used for the web server itself. Flask and Gunicorn were used to host and run the server.

capture_extract: -capture.py - Capture incoming traffic to the web server. Create new .pcap for each new source IP with a timeout of 10 minutes. -cap_with_len_csv.sh - Convert output .pcap to CSV for analysis. -burstiness.py - Calculate burst metrics from input session. -duration.py - Calculate session duration metrics from input session. -interarrival.py - Calculate interarrival time metrics from input session. -packet_size.py - Calculate packet size metrics from input session. -all_analysis.py - Run all 4 analysis scripts. -dataset.py - Combine the output of all analysis scripts into a single CSV then move all component files into a subdirectory based on the initial capture name.

analyze_compare: -group.py - Short script to combine all output CSVs from dataset.py into a single master dataset for the given group. -comparisons.py - Run comparison analyses between the 3 master dataset files to calculate KS values, t-test values, and Cohen's d, provide analysis of values, and generate KDE charts of all values. -pca_plot.py - Generate PCA plot of top values from analysis with vector arrows showing value influence on plotting.

bots: -agentic_selenium.py - Selenium-based headful agentic bot running pre-determined actions with variable timing. -browseruse_llm.py - BrowserUse backend using Gemini LLM to navigate the site with human-like instructions. -crawler.py - BeautifulSoup crawler. -IWR.ps - PowerShell Invoke-WebRequest crawler. -openai_llm.py - BrowserUse backend using OpenAI LLM to navigate the site with human-like instructions. -playwright-agentic.py - Playwright-based headful agentic bot running pre-determined actions with variable timing. -playwright-headless.py - Playwright-based headless agentic bot running pre-determined actions with variable timing. puppeteer_headless.js - Puppeteer-based headless agentic bot running pre-determind actions with variable timing. -wget_crawl.sh - Basic wget shell script to crawl the page.

All Python packages used for capture, extraction, and analysis stored in requirements.txt.

Please contact William Hatfield at william.e.hatfield@gmail.com with any questions.