OpenML EvaluationEngine (Python)

Python port of the OpenML EvaluationEngine — the service that processes datasets (features & meta-features), generates folds, merges multitask datasets, and evaluates submitted runs on the OpenML server.

What it does

Function (-f)	Java original	What it does
evaluate_run	EvaluateRun	Download a run's predictions/splits/dataset, compute all evaluation scores, upload them (--no-upload prints the XML locally)
process_dataset	ProcessDataset	Extract + upload dataset features and qualities; --id for one dataset, omit to poll for unprocessed ones
process_dataset_print	processAndPrint	Extract features locally and print the XML (no upload)
extract_features_simple	FantailConnector (simple)	Compute + upload base qualities (SimpleMetaFeatures + pymfe groups); polls when no --id
extract_features_all	FantailConnector (all)	Same, plus sklearn landmarker meta-features
generate_folds	GenerateFolds	Generate the splits table (cross-validation, holdout, learning-curve, ...) for a task and write it as ARFF
merge_datasets	MergeDataset	Merge all datasets of a MultiTask task into one ARFF
all_wrong, different_predictions, challenge	—	Not ported (see src/main.py docstring)

Setup

Requires Python >= 3.13 and uv.

uv sync                # create .venv and install dependencies
uv run pytest          # run the test suite (offline; ~400 tests)

API authentication: set OPENML_API_KEY in the environment, or rely on the test server's default demo key when using --test (below).

CLI usage

Entry point: python -m src.main (-f/--function is required; run with --help for all flags).

# Evaluate one run on the test server (test.openml.org, demo API key)
uv run python -m src.main -f evaluate_run -id 1 -test

# Same, but only compute and print the evaluation XML without uploading
uv run python -m src.main -f evaluate_run -id 1 -test -no-upload

# Process a single dataset / start the processing poll loop
uv run python -m src.main -f process_dataset -id 61 -test
uv run python -m src.main -f process_dataset -test

# Print a dataset's features as XML (no upload)
uv run python -m src.main -f process_dataset_print -id 61 -test

# Generate the folds ARFF for a task (writes to --output or stdout)
uv run python -m src.main -f generate_folds -id 1720 -test -o splits.arff

# Meta-features (qualities) for one dataset, or poll the server
uv run python -m src.main -f extract_features_simple -id 61 -test
uv run python -m src.main -f extract_features_all -test

# Merge the datasets of a MultiTask task
uv run python -m src.main -f merge_datasets -id 1901 -test -o merged.arff

Useful flags:

Flag	Meaning
-test	Target test.openml.org (default key normaluser) instead of production
-id	Dataset / run / task id, depending on the function
-df	Dataset file format: arff (default) or parquet
-no-upload	Compute locally, skip uploading (evaluate_run prints the XML)
-o/--output	Output file (generate_folds, merge_datasets)
-tag, -mode, -u, -t, -x, -reverse	Priority tag, mode, uploader/task filters, random/reverse ordering

Docker

The image runs the same CLI as its entrypoint:

# Build
docker build -t evaluation-engine .

# Prints usage
docker run --rm evaluation-engine

# Evaluate a run on the test server
docker run --rm evaluation-engine -f evaluate_run -id 1 -test

# Dry run (no upload), mounting an output directory for generated files
docker run --rm -v "$PWD/out:/out" evaluation-engine \
  -f generate_folds -id 1720 -test -o /out/splits.arff

# Production server: pass your API key through the environment
docker run --rm -e OPENML_API_KEY evaluation-engine -f process_dataset -id 61

The image installs only runtime dependencies (uv sync --no-dev) and does not contain the test suite.

Documentation

API reference and usage docs are built with Zensical (successor of MkDocs 1.x + Material, which our config stays compatible with):

uv run zensical serve    # live preview at http://localhost:8000
uv run zensical build    # static site in site/

This README is the docs home page; the full API reference lives under docs/api/ (see mkdocs.yml). The site is built and deployed to GitHub Pages by the Docs workflow on every push to main.

Project layout

src/
├── main.py               # CLI dispatcher (Main.java)
├── client.py             # OpenML REST client (OpenmlConnector subset)
├── evaluate_run.py       # run evaluation driver (EvaluateRun.java)
├── helpers.py            # HTTP/XML + ARFF + prediction math helpers
├── data_loader.py        # ARFF/Parquet -> normalized (attributes, rows)
├── models.py             # dataclasses shared across the engine
├── exceptions.py         # OpenmlError hierarchy
├── runs/                 # run evaluation: metrics, fold counting, XML
├── features/             # dataset feature extraction + XML
├── qualities/            # meta-features (pymfe), landmarkers, polling
└── process_dataset/      # fold generation, dataset processing, merging
tests/                    # pytest suite mirroring src/ layout
docs/                     # MkDocs sources (API reference)
reference_and_sanity_checks/   # manual parity notebooks vs the Java engine

Parity notes

The Python engine mirrors the Java behaviour (metric definitions, seed usage, error-code handling such as 431/441), with documented deviations where a Weka component has no Python equivalent — e.g. landmarker classifiers are sklearn approximations and the CfsSubsetEval_* meta-features are omitted. The reference_and_sanity_checks/ notebooks compare outputs against the live test server.