IC-ThermBench

An open benchmark for reproducible, generalizable 2.5D/3D-IC thermal learning.

Paper · Quick start · Datasets · Results · Reproduce · Add a model · Paper preview

Thermal prediction papers often differ in data, simulators, splits, preprocessing, and metrics, making model-to-model comparison surprisingly fragile. Moreover, most existing datasets and implementations are not publicly available, and the community still lacks a unified evaluation standard. IC-ThermBench fixes that evaluation contract. It provides progressive physical support, immutable splits, six methods represented by eight configurations across three architecture families, and one interface for training, inference, adaptation, and reporting.

Developed and maintained by the IC-ThermBench research team at the University of Technology Sydney (UTS).

Why IC-ThermBench

Recent methods study geometry, material, cooling, and unseen systems. The remaining problem is comparability. Results are commonly produced with different private or partially released datasets, reference solvers, representations, splits, and metrics. Consequently, a lower error may reflect an easier test distribution rather than a better model, and “generalization” may refer to anything from a new power map on the same chip to a structurally unseen package.

G/P/M/B/t denote geometry or placement, power, material, boundary or cooling, and time. “Closest Scope” is an approximate capability mapping, not a claim that the datasets or splits are equivalent. Citation indices follow the manuscript bibliography.

What the benchmark reveals

Track	Physical support	Best method	Result ↓	Second-best method	Result ↓
S1	fixed-design source tasks	Therm-FM	0.004–0.049 K MAE¹	—	—²
S2	represented layouts and configurations	Therm-FM	0.443 K RMSE	SAU-FNO	0.703 K RMSE
S3	S2 + material conductivity	Therm-FM	0.716 K RMSE	U-FNO	0.802 K RMSE
S4	S3 + ambient and cooling conditions	Therm-FM	0.933 K RMSE	SAU-FNO	1.216 K RMSE
S5 · zero-shot	five case-disjoint chiplet systems	Therm-FM	15.51 K RMSE	U-Net	19.10 K RMSE
S5 · 10-shot	ten labeled samples per unseen case	Therm-FM	2.73 K RMSE	U-FNO	3.59 K RMSE

¹ S1 contains eleven source tasks, so the paper reports task-group MAE ranges rather than one pooled score. ² Its second-best method varies by task and resolution, so no single runner-up is reported. Therm-FM T/B/L are ranked as one method family, using the best variant score on each track. The selected benchmark results include the S1 source-suite summary, compact S2–S4 comparisons, and the main S5 zero-shot and few-shot findings. S2–S5 use the controlled IC-ThermBench protocol. Best results are bold; second-best results are underlined.

The best in-support RMSE rises from 0.443 K on S2 to 0.716 K on S3 and 0.933 K on S4 as observed physical dimensions are added. The case-disjoint S5 shift is qualitatively different: the best RMSE grows by about 16.6×, model rankings change, and a small amount of target supervision recovers much of the gap. See the paper preview for interpretation and the reproduction guide for the exact protocol.

Benchmark at a glance

IC-ThermBench uses Scope rather than “level”: the sequence describes the capability a deployment needs, not a universal maturity rank.

Scope	Evaluation setting	Newly variable factors	Samples	Typical use
S1 · Source suite	fixed-design prediction	power; time for transient tasks	32,000	workload analysis, dynamic thermal management
S2 · Layout	in-support	placement and orientation across system templates	15,000	floorplanning and design-space exploration
S3 · Material	in-support	S2 + local thermal conductivity	15,000	material and process sweeps
S4 · Boundary	in-support	S3 + ambient temperature and convection	15,000	cooling and environment co-design
S5 · Structural OOD	case-disjoint	unseen chiplet counts, sizes, power-density regimes, and utilization	5,000	transfer to a new product or package

Dataset provenance and credit

• S1 is collected, not regenerated. IC-ThermBench preserves the original Alpha EV6 and industrial task definitions, simulators, resolutions, and evaluation conventions used along the ARO → Therm-FM research line. We do not alter these source tasks or pool their scores with S2–S5. The S1 source-task datasets are now released; the unified one-command evaluator is on the way, and the recorded benchmark results are available now.
• S2 follows the ATPlace2.5D lineage. Cases 1–10, their chiplet systems, and the HotSpot-based thermal setup originate from Qipan Wang et al.'s ATPlace2.5D public package. IC-ThermBench uses this tested foundation for its layout scope.
• S3–S5 are consistent extensions. They retain the S2 case/generation conventions while adding material support, boundary support, and held-out structural systems. This preserves continuity with an established placement benchmark while expanding the thermal-learning evaluation space.

The current executable release contains the generator-backed S2–S5 data and code path. The S1 datasets are released separately and kept distinct from S2–S5; a unified S1 evaluator is on the way.

Six methods, one protocol

Architecture family	Methods / configurations
Convolutional networks	U-Net
Neural operators	FNO, U-FNO, SAU-FNO, DeepOHeat
PDE foundation models	Therm-FM T / B / L

Every baseline receives the same labeled samples, split membership, physical channels, and metric implementation. The release does not force one optimizer onto every architecture: model-specific training recipes are preserved in MODEL_ZOO.

For DeepOHeat, we report fully supervised training rather than its label-free physics-residual scheme so that it is compared fairly with the other supervised baselines. In our tests, the supervised variant was also substantially more accurate and more stable to train.

Quick start

1. Install

git clone https://github.com/Day333/ThermalBench.git
cd ThermalBench
conda env create -f environment.yml
conda activate ic-thermbench
python script/smoke_test.py

The frozen environment matches the released checkpoints. Therm-FM is version-sensitive; read the installation notes before changing PyTorch, Transformers, or Accelerate.

2. Place data and checkpoints

Download the datasets (~4.6 GB) and released checkpoints (~44 GB), then unpack them at the repository root:

IC-ThermBench/
├── datasets/
│   ├── level2_steady/
│   ├── level3_steady/
│   ├── level4_steady/
│   └── level5_steady/
└── checkpoints/

The public command tokens remain level2–level5 for checkpoint compatibility; they correspond to paper Scopes S2–S5.

3. Evaluate

# One baseline on one scope
python run.py --model UFNO --data level2 --task test

# Frozen S4 checkpoint on structural-OOD S5
python run.py --model ThermFM-T --data level5 --task test

# All 8 evaluated configurations × S2–S5, followed by one summary table
bash script/test_all.sh

When --load is omitted, S5 zero-shot evaluation automatically resolves the matching S4 checkpoint. No S5 labels or normalization updates are used.

Train, adapt, compare

# Train one model on S2/S3/S4
bash script/UFNO/train.sh

# Ten labeled samples per unseen S5 case
python run.py --model UFNO --data level5 --task finetune --shots 10

# Run all released adaptation recipes
bash script/finetune_all.sh

# Rebuild the benchmark summary from saved metrics
python utils/summarize.py level2 level3 level4 level5

run.py is the single entry point; every command also accepts --root_path, --checkpoints, --load, and --output. Full commands, split semantics, metric definitions, and reproducibility boundaries are in docs/REPRODUCE.md.

Use IC-ThermBench for your model

A new predictor only needs to satisfy one tensor contract:

input   (B, X, Y, Z, P)
output  (B, X, Y, Z)       where X = Y = 64 and Z = 1

Register the constructor and training recipe once, then the existing scripts provide matched S2–S5 training, zero-shot evaluation, few-shot adaptation, and the shared metrics. The complete three-step example is in docs/ADD_A_MODEL.md.

Reproducibility contract

• Data: published tensors, physical-channel schema, and S5 case manifest.
• Splits: deterministic, index-based train/validation/test membership.
• Labels: supervised reference-temperature fields for every baseline.
• Metrics: one implementation for RMSE, MAE, R², MaxAE, peak-temperature error, and Top-50 MAE.
• Recipes: architecture-specific optimization is recorded, not retroactively homogenized.
• Artifacts: released checkpoints and JSON outputs can be audited without retraining.

The release is designed to be easy to reproduce and easy to extend: data conversion, model execution, evaluation, aggregation, and OOD adaptation all use the same repository-level interface.

Documentation

Guide	Contents
Installation	exact environment, GPU notes, Therm-FM dependencies
Datasets	files, shapes, channels, splits, manifests, checkpoints
Benchmark results	S1 source tasks, S2–S4 in-support results, S5 zero-shot and adaptation
Reproduce	evaluation tracks, commands, metrics, expected key results
Add a model	tensor contract, registry entry, scripts, integration checklist
Paper preview	benchmark motivation, design, findings, and paper status

Current limitations

• S1 datasets are released; a unified one-command evaluation script is still in progress, and its stored values currently preserve source-reported protocols.
• S2–S5 currently cover 64×64 steady-state fields with Z=1; progressive transient evaluation beyond S1 remains future work.
• S2–S4 are independently generated, not sample-wise paired, so their differences measure distributional difficulty rather than a strict causal ablation.
• S5 contains five held-out systems. It reveals a substantial structural-OOD gap, but cannot represent every industrial package, process, or cooling technology.
• The benchmark is label-supervised. Physics-only and label-free training answer a different question and are not presented as directly interchangeable baselines.

Citation

If you use IC-ThermBench, please cite the paper.

@article{icthermbench2026,
  title   = {IC-ThermBench: An Open, Progressive Benchmark for Generalizable
             2.5D/3D-IC Thermal Learning},
  author  = {David Hang and Wenkai Yang and Kuiye Ding and Haiyang Xin and Jacky Wei},
  journal = {arXiv preprint arXiv:2608.23977},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.23977}
}

License and acknowledgements

• Repository code is released under the MIT License.
• Original IC-ThermBench S2–S5 data, fixed splits, and released result records are available under CC BY 4.0: reuse and extension are welcome with attribution.
• S1 source data and third-party components retain their original licenses. The IC-ThermBench manuscript and paper-derived prose, figures, and tables are not relicensed as dataset content.

See the license map and attribution guidance for the exact boundaries. Open licensing permits legitimate reuse; it does not permit claiming the benchmark, paper text, or curation work as someone else's contribution.

We thank the authors of ARO, Therm-FM, and ATPlace2.5D, as well as the HotSpot, Poseidon/scOT, FNO, U-FNO, and DeepOHeat communities. Their public artifacts and source tasks make a shared thermal-learning benchmark possible.