Data Platform Sample

Architecture

From the diagram, users can query the data through Trino. Trino communicates with Apache Polaris through the REST Catalog API to retrieve Iceberg table metadata, while the actual data is stored in MinIO. On the other hand, the Airflow orchestrated data pipeline extracts data from external sources and writes it to MinIO as Iceberg tables, using Apache Polaris as the catalog through the same REST Catalog API.

How to setup and deploy

Prerequisites

• Minikube: >= v1.38.1
• kubectl: >= v1.36.3
• Helm: >= v4.2.2

Setup and Deployment

1. Extract the project files.
2. Start Minikube with the required resources:

   minikube start --memory=8g --cpus=4

3. Run the installation script to set up and deploy the platform:

   ./install.sh install

The install.sh script provides four options:

Command	Description
./install.sh install	Install and configure the platform, making it ready to use.
./install.sh uninstall	Remove the platform and clean up its Kubernetes resources.
./install.sh upgrade	Upgrade the deployment when Helm charts or persistent volumes have changed.
./install.sh update-dags	Update the Airflow DAGs when DAG definitions have changed.

📍Important: Running ./install.sh install will clean up the “data-platform”, “external-resources”, and “operators” namespaces and remove their existing Kubernetes resources, including Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, Services, ConfigMaps, Secrets, PVCs, Ingresses, ServiceAccounts, Roles, and RoleBindings. This ensures a clean state and helps prevent conflicts or errors caused by leftover resources from previous deployments.

Manual Setup

Manual setup is also useful for troubleshooting. If ./install.sh install fails, you can run each step manually to identify which component causes the error.

1. Build the Helm dependencies

   helm dependency build ./operators
   helm dependency build ./external-resources
   helm dependency build ./data-platform

2. Install the operators

   helm install operators ./operators \
     --namespace data-platform \
     --create-namespace

3. Install the external resources

   helm install external-resources ./external-resources \
     --namespace external-resources \
     --create-namespace

4. Build the custom Airflow image

   docker build -t dag-airflow:latest .

5. Load the Airflow image into Minikube

   minikube image load dag-airflow:latest

6. Install the data platform

   helm install data-platform ./data-platform \
     --namespace data-platform