Shopify Store Replicator - MVP

A Python tool to replicate products from any Shopify store to your target store using modern APIs (Firecrawl, Gemini 2.5 Flash, and Shopify Admin API).

Features

• 🔍 Smart URL Extraction: Multiple strategies to find product URLs from sitemaps
• 🕷️ Advanced Scraping: Uses Firecrawl for reliable content extraction
• 🧠 AI Processing: Gemini 2.5 Flash structures data into Shopify format
• 📤 Shopify Integration: Direct upload to your Shopify store
• 🏃 Dry Run Mode: Test without uploading
• 📊 Progress Tracking: Real-time progress bars and detailed logging

Quick Start

1. Setup Environment

# Clone or download the project
cd replicate-store

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Configure API Keys

Copy the environment template and add your API keys:

cp .env.example .env
# Edit .env with your API keys

Required API keys:

• Firecrawl API: Get from firecrawl.dev
• Gemini API: Get from Google AI Studio
• Shopify Access Token: Create from your Shopify admin

3. Run the Tool

# Basic usage
python run.py example-store.myshopify.com

# Dry run (process but don't upload)
python run.py example-store.myshopify.com --dry-run

# Limit products and enable verbose logging
python run.py example-store.myshopify.com --max-products 20 --verbose

Configuration

Environment Variables (.env)

# Required API Keys
FIRECRAWL_API_KEY=fc-your-api-key-here
GEMINI_API_KEY=your-gemini-api-key-here
SHOPIFY_STORE_URL=https://your-store.myshopify.com
SHOPIFY_ACCESS_TOKEN=your-access-token-here

Settings (config/settings.yaml)

# Maximum number of products to process
max_products: 50

# Scraping settings
scraping_timeout: 30
requests_per_minute: 30

# Processing settings
gemini_model: "gemini-2.5-flash"

# Upload settings
batch_size: 5

How It Works

Stage 1: URL Extraction 🔍

• Fetches sitemap.xml from source store
• Tries multiple strategies to find product URLs
• Filters and validates product links

Stage 2: Content Scraping 🕷️

• Uses Firecrawl to scrape product pages
• Extracts both markdown and HTML content
• Handles rate limiting automatically

Stage 3: AI Processing 🧠

• Sends content to Gemini 2.5 Flash
• Structures data into Shopify product format
• Validates extracted information

Stage 4: Shopify Upload 📤

• Creates products in your Shopify store
• Handles variants, images, and metadata
• Processes in batches with rate limiting

Command Line Options

python run.py <source-domain> [options]

Arguments:
  source_domain         Source Shopify domain (e.g., example.myshopify.com)

Options:
  --dry-run            Process but don't upload to Shopify
  --max-products N     Maximum products to process (overrides config)
  --verbose, -v        Enable verbose logging
  --help, -h           Show help message

Output

Logs

• Console output with progress bars
• Detailed logs saved to logs/ directory
• Error tracking and debugging info

Results

• Processed products saved to results/ directory
• JSON format for review and backup
• Includes extraction timestamp and stats

API Requirements

Shopify Admin API

Your Shopify app needs these permissions:

• read_products
• write_products
• read_product_listings
• write_product_listings

Rate Limits

• Firecrawl: 30 requests/minute (configurable)
• Gemini: Built-in retry with exponential backoff
• Shopify: 2 requests/second (handled automatically)

Troubleshooting

Common Issues

"No product URLs found"

• Check if the source domain is a valid Shopify store
• Some stores may have restricted sitemaps
• Try with a different store

"Failed to scrape products"

• Verify your Firecrawl API key
• Check if you have sufficient credits
• Some pages may be protected

"Gemini processing failed"

• Verify your Gemini API key
• Check if you have API quota remaining
• Content may be too large (automatically truncated)

"Shopify upload failed"

• Verify your Shopify credentials
• Check API permissions
• Review product data validation errors

Debug Mode

Enable verbose logging for detailed debugging:

python run.py example-store.myshopify.com --verbose --dry-run

Limitations (MVP Version)

• Images: Uses original URLs (doesn't download/re-upload)
• Collections: Doesn't replicate collection structure
• SEO: Doesn't preserve meta descriptions or URL structure
• Inventory: Sets basic inventory levels
• Variants: Basic variant extraction

Project Structure

replicate-store/
├── src/
│   ├── config.py              # Configuration management
│   ├── sitemap_extractor.py   # URL extraction from sitemaps
│   ├── firecrawl_scraper.py   # Content scraping
│   ├── gemini_processor.py    # AI data processing
│   ├── shopify_uploader.py    # Shopify integration
│   ├── utils.py               # Utility functions
│   └── main.py                # Main orchestrator
├── config/
│   └── settings.yaml          # Configuration file
├── logs/                      # Log files
├── results/                   # Processed products
├── requirements.txt           # Dependencies
├── .env.example              # Environment template
├── run.py                    # Entry point
└── README.md                 # This file

Legal & Ethical Use

• Only replicate stores you own or have permission to replicate
• Respect robots.txt and terms of service
• Use for legitimate business purposes only
• Consider data privacy and copyright implications

Support

For issues and questions:

1. Check the troubleshooting section above
2. Review logs in the logs/ directory
3. Test with --dry-run and --verbose flags
4. Verify all API keys and permissions

---

Note: This is an MVP version focused on core functionality. Future versions may include advanced features like image optimization, collection replication, and incremental updates.