At PyConDE 2022, we spent several hours with the TV screen showing a hanged cluster and we had to debug the issue in realtime, while embarassingly standing at the stall.
The problem was caused by a regression in the very latest dask version, released two days before, which was immediately picked up by the examples because the packages in the conda environment weren't pinned.
Everything should be pinned to prevent this kind of surprises. Upgrades should be a conscious act that includes retesting.
At PyConDE 2022, we spent several hours with the TV screen showing a hanged cluster and we had to debug the issue in realtime, while embarassingly standing at the stall.
The problem was caused by a regression in the very latest dask version, released two days before, which was immediately picked up by the examples because the packages in the conda environment weren't pinned.
Everything should be pinned to prevent this kind of surprises. Upgrades should be a conscious act that includes retesting.