use 8 cores for building hipBLASLt and rocBLAS - #302
Conversation
|
bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
|
New job on instance
|
|
New job on instance
|
|
Still testing this (using the commit from this PR) in EESSI/software-layer#1549. |
OK, let us know if it works as expected/hoped... |
…r-scripts into hipblast_max_cores
|
bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
|
New job on instance
|
|
New job on instance
|
hipBLASLthipBLASLt and rocBLAS
|
@bedroge Can you update the PR description a bit to explain why this is necessary? There was a lot of back & forth, but it the end it's a pretty clean change. It does still require |
Done! |
|
bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace |
|
New job on instance
|
|
New job on instance
|
|
New job on instance
|
|
@ocaisa Why the rebuild? |
|
Sorry, I thought the updated comment was in the easystack itself, didn't check properly if there was a commit |
|
bot:cancel jobid:15623810 |
|
bot:cancel jobid:15623811 |
|
bot:cancel jobid:15623812 |
|
staging PR merged |
Both hipBLASLt and rocBLAS are crashing nodes, seemingly due to the excessive number of threads and/or due to memory usage. Limiting this to 8, and making sure that Tensile also only uses 8 threads (see easybuilders/easybuild-easyconfigs#26846, something similar was already done for hipBLASLt: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/h/hipBLASLt/hipBLASLt-0.12.1-rocm-compilers-19.0.0-ROCm-6.4.1.eb#L66C1-L66C52) solves the issues.
Do note that these builds require a lot of memory (>50-60GB or even more) for certain GFX targets (in particular gfx942), so they have to be done on larger build nodes. For this we temporarily switched the bot configs to one that contains larger build nodes (
-xlnodes), restarted the event handler, submitted the jobs, and reverted back to the original config.