pywikiscrape

pywikiscrape is a user-friendly but powerful open-source Wikipedia scraper written in Python that lets users export articles in any language to an SQLite3 database using Beautiful Soup.

pywikiscrape now supports all Wikipedias, English and non-English, all with manual and automatic seeds

---

Features

• Automated Scraping: Automatically scrapes Wikipedia articles with requests and BeautifulSoup.
• Supports Non-English Wikipedias: Supports scraping for all English and non-English Wikipedias, along with automatic and manual seeds.
• Minimal-Config: Simply run the script, enter a couple of settings or click enter for defaults, and sit back and relax.
• Two SQLite3 Tables: Stores two tables, one for the content of the article, the other for the links contained in it, in one file.
• Minimal Dependencies: Relying only on Python standard library modules, requests, and BeautifulSoup.
• Error Handling: Handles many types of errors on its own, and only asks for extra user input if necessary.
• Logging: Uses the Python Standard Library module logging over print statements.

Usage

Install with pip or pipx (recommended)

It is recommended for Linux and macOS users to use pipx rather than pip due to the high possibility of your Python being marked as externally managed in order to prevent the overwriting of files needed by your operating system, which can break system tools. Windows users are free to use pip or pipx to install pywikiscrape.

Tool	Command
pip	pip install pywikiscrape
pipx	pipx install pywikiscrape

Clone repository and manually run

If you wish to contribute to this project, you must clone this repository to your local machine, make your changes, and make a pull request. To do so, follow this code block. It is recommended for non-Windows users to use pipx instead of pip to install the dependencies, but either may be used if your operating system or distribution allows so.

# Clone the repository
git clone https://github.com/tyleruploads/pywikiscrape.git
cd pywikiscrape

# Set up a virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pipx install -r requirements.txt

# Runs the script
python3 src/main.py

The script saves its output to an SQLite3 database with the default name of LANGUAGE_CODE-pywikiscrape.db. All required variables are extracted from the user via text prompts. There are no arguments you need to pass to the program.

The seed can either be chosen manually by the user or automatically by the script.

Database Schemas

Table 1: Articles

This table contains the title of the Wikipedia article and the text inside of it.

Column	Type	Description
id	INTEGER	Primary Key
title	TEXT	Wikipedia article Title
text	TEXT	Content of Wikipedia article

Table 2: Links

This table contains the id correlating an article to its entry in table 1 and the links contained inside of it.

Column	Type	Description
id	INTEGER	Primary Key
links_json	TEXT	The links contained in the Wikipedia article

Roadmap

• Package and publish to PyPI
• Have the CLI ask the user for a seed, a custom path for the output database, and other settings, all with defaults the user can easily select
• Support scraping non-English Wikipedias
• Store more metadata about article like categories, images, and when it was added to the database
• Allow more export options for data like JSON, CSV, or Parquet alongside SQLite
• Add Unit testing
• Add a Dockerfile and docker-compose.yml for zero-setup execution

Contributing

Contributions are what make open-source projects one of a kind. All contributions are highly appreciated.

• Found a bug or issue: Open an Issue and show the output of the script, the steps to reproduce it, and as much information as possible
• Have an idea: Open an Issue and explain your idea as much as possible, why you think it would be a good addition to the project, and any other important information.

License

This project is licensed under the MIT License - see the LICENSE file for more information.