FootballIQ ⚽📊

FootballIQ is an end-to-end, high-performance football data engineering pipeline. It ingests, processes, stores, and serves live and historical football data from the SportMonks API using a modern distributed data stack.

---

🏗️ Architecture Overview

FootballIQ follows Clean Architecture principles to separate business logic, data pipelines, infrastructure, and API layers:

FootballIQ/
├── docker/                 # Infrastructure setup (Kafka, Zookeeper, MongoDB, Redis, Airflow)
├── src/
│   ├── api/                # Presentation layer (FastAPI endpoints)
│   ├── config/             # Environment & settings configuration (Pydantic Settings)
│   ├── dags/               # Apache Airflow orchestration DAGs
│   ├── domain/             # Core business models (Pydantic v2 schemas: Match, Team, Player, etc.)
│   ├── Ingestion/          # Data ingestion pipelines (Batch & Streaming)
│   └── storage/            # Infrastructure data access (MongoDB repository & Redis cache)
└── requirements.txt        # Python dependencies

---

🔄 Data Pipeline Flow

1. Ingestion Layer (src/Ingestion)

   • Streaming: Consumes live match events from SportMonks API and streams them into Apache Kafka with snappy compression and idempotent producers.
   • Batch: Fetches historical seasons, leagues, fixtures, and standings orchestrated via Apache Airflow.

2. Storage Layer (src/storage)

   • MongoDB: Serves as the primary document database for raw and structured football entities (fixtures, teams, standings, player statistics).
   • Redis: Provides high-speed in-memory caching to optimize API response latency.

3. Presentation Layer (src/api)

   • FastAPI: Exposes RESTful endpoints for querying match data, standings, and player metrics.

---

🚀 Tech Stack

• Language: Python 3.10+
• API Framework: FastAPI, Pydantic (v2)
• Streaming & Messaging: Apache Kafka, Confluent Kafka SDK
• Orchestration: Apache Airflow (v3 TaskFlow API)
• Databases: MongoDB (pymongo), Redis (redis-py)
• Testing: Pytest
• Containerization: Docker, Docker Compose

---

⚙️ Prerequisites & Setup

1. Clone & Environment Configuration

Create a .env file in the project root:

SPORTMONKS_API_KEY=your_api_key_here
MONGO_URI=mongodb://localhost:27017
REDIS_HOST=localhost
REDIS_PORT=6379
KAFKA_BOOTSTRAP_SERVERS=localhost:9092

2. Infrastructure Deployment (Docker Stack)

Launch the full infrastructure services (Kafka, Zookeeper, MongoDB, Redis, Airflow):

docker-compose -f docker/docker-compose.yml up -d

3. Python Environment Setup

python -m venv venv
# On Windows:
venv\Scripts\activate
# On Linux/macOS:
source venv/bin/activate

pip install -r requirements.txt

---

🧪 Testing & Verification

Run integration and unit tests using pytest:

pytest src/test/ -v

---

📡 API Endpoints

Launch the API server:

uvicorn src.api.Get_Endpoint:app --reload

Interactive API documentation available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

---

📜 License

This project is licensed under the MIT License.