Huge performace drop when using DeepSpeed-ZeRO2 on 4 A40 GPUs.

[twelfth-star]

Thank you for your impressive work!

I tried to use 4 A40 GPUs (48GB each) to reproduce your results. Due to the memory limit, I applied DeepSpeed-ZeRO2. However, the performance on APPS and livecodebench dropped significantly. Do you have any suggestions?

[wanglingxiang0717]

Thank you for your impressive work!

I tried to use 4 A40 GPUs (48GB each) to reproduce your results. Due to the memory limit, I applied DeepSpeed-ZeRO2. However, the performance on APPS and livecodebench dropped significantly. Do you have any suggestions?

I'm encountering almost the same issue. Have you found a solution yet? Could it be caused by DeepSpeed?