Skip to content

ec2_conversion.md

Latest commit

 

History

History
171 lines (124 loc) · 4.99 KB

ec2_conversion.md

File metadata and controls

171 lines (124 loc) · 4.99 KB
Raw

EC2 conversion runbook — MPD JSON → Parquet on a single box

Convert the Spotify Million Playlist Dataset to Parquet on one large EC2 instance with single-node engines. No EMR, no Spark, no cluster.

The winning shape is a compute instance with local NVMe instance storage (e.g. m6id.8xlarge — 32 vCPU / 128 GB RAM / 1.9 TB NVMe). Local NVMe removes the disk-throughput ceiling that gp3 EBS imposes (see benchmark_results.md).

Placeholders below: <BUCKET> = your S3 bucket, <PUBLIC_IP> = the instance's public IP, <KEY>.pem = your SSH key.

1. Launch the instance

  • AMI: Amazon Linux 2023
  • Instance type: m6id.8xlarge (local NVMe). A cheaper m6a.8xlarge with gp3 EBS also works but is I/O-bound at ~500 MB/s.
  • IAM instance profile: a role with AmazonS3ReadOnlyAccess (to pull the zip) + write access to your output bucket/prefix.
  • Root EBS: ~30 GB is plenty — the dataset lives on the NVMe, not root.
  • Security group: allow SSH (port 22) from your IP.

Fix key permissions and connect:

chmod 600 <KEY>.pem                       # Windows: icacls <KEY>.pem /inheritance:r /grant:r "%USERNAME%:R"
ssh -i <KEY>.pem ec2-user@<PUBLIC_IP>

Keep an S3 gateway endpoint on the VPC so EC2 ⇄ S3 traffic stays in-region and free.

2. Format and mount the local NVMe at /data

The instance-store NVMe shows up as an unmounted block device:

lsblk -d -o NAME,SIZE,MODEL          # find the ~1.9 TB instance-store device, e.g. nvme1n1
sudo mkfs.xfs -f -q /dev/nvme1n1
sudo mkdir -p /data
sudo mount /dev/nvme1n1 /data
sudo chown ec2-user:ec2-user /data
df -h /data

(Device name varies; on some AMIs the instance store is nvme0n1 and the root EBS is nvme1n1 — check lsblk output.)

3. Install Python 3.11 and the engines

AL2023 ships Python 3.9; DuckDB/polars want ≥ 3.10, so install 3.11:

sudo yum install -y -q unzip htop python3.11 python3.11-pip
python3.11 -m venv ~/venv
~/venv/bin/pip install -q --upgrade pip
~/venv/bin/pip install -q "duckdb==1.5.4" "polars==1.42.1" "pyarrow>=24"
~/venv/bin/python -c "import duckdb,pyarrow,polars; print(duckdb.__version__, pyarrow.__version__, polars.__version__)"

Copy the three converter scripts to the box (scp from src/, or clone this repo).

4. Stage the data on NVMe

Download the zip from S3 straight onto /data and unzip the slices into data1/:

cd /data
aws s3 cp s3://<BUCKET>/spotify_million_playlist_dataset.zip . --no-progress
unzip -q spotify_million_playlist_dataset.zip -d unz
mv unz/data data1 && rmdir unz
ls data1/*.json | wc -l          # ~1000 slices, ~30 GB

To benchmark at volume, make real copies into more folders. Use --reflink=never — XFS reflink makes copy-on-write clones that share blocks, which gives later folders an unfair page-cache advantage and skews timings:

# 8 folders total (data1..data8) = ~256 GB, ~8M playlists
for n in 2 3 4 5 6 7 8; do cp -r --reflink=never data1 data$n; done
df -h /data

5. Run a converter

DuckDB (fastest — streaming, ~7 GB RAM), one worker per folder, 8 at once, ~128 MB output files:

cd /data
~/venv/bin/python ~/json_to_parquet.py --per-folder --file-size-mb 128 \
    --max-concurrent 8 \
    --data-dir data1 data2 data3 data4 data5 data6 data7 data8 \
    --out-dir /data/parq --temp-dir /data/dtmp --memory-limit-gb 100

PyArrow:

~/venv/bin/python ~/json_to_parquet_pyarrow.py --per-folder --max-concurrent 8 \
    --data-dir data1 data2 data3 data4 data5 data6 data7 data8 \
    --out-dir /data/parq_pa

polars (materializes each folder — cap concurrency to stay under RAM; 8-wide OOMs on 128 GB, 6-wide fits):

~/venv/bin/python ~/json_to_parquet_polars.py --per-folder --max-concurrent 6 \
    --data-dir data1 data2 data3 data4 data5 data6 data7 data8 \
    --out-dir /data/parq_pl

--max-concurrent sets how many folders run at once; the memory budget and CPU cores are split across that many workers. Lower it if a worker OOMs.

Watch resource use in another SSH session with htop — on NVMe you should see all cores near 100% and only a few GB of RAM in use (DuckDB/PyArrow); if cores sit idle at 10–25% you are disk-bound (that's the gp3 case).

6. Push the Parquet output to S3

aws configure set default.s3.max_concurrent_requests 32
aws s3 sync /data/parq s3://<BUCKET>/output/parq/ --no-progress

7. Verify

Row count across the output (expect 8,000,000 for 8 folders — one row per playlist):

~/venv/bin/python - <<'PY'
import duckdb
c = duckdb.connect()
n = c.execute("SELECT count(*) FROM read_parquet('/data/parq/*/*.parquet')").fetchone()[0]
print(f"{n:,} rows")
PY

8. Terminate the instance

Instance-store data is lost on stop/terminate — make sure the S3 sync finished first.

aws ec2 terminate-instances --region us-east-1 --instance-ids <INSTANCE_ID>

The only ongoing charge is the Parquet you left in S3 (~$0.023/GB-month).