Skip to content

use 8 cores for building hipBLASLt and rocBLAS - #302

Merged
boegel merged 11 commits into
EESSI:mainfrom
bedroge:hipblast_max_cores
Sep 15, 2026
Merged

boegel merged 11 commits into
EESSI:mainfrom
bedroge:hipblast_max_cores

Conversation

@bedroge

@bedroge bedroge commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Both hipBLASLt and rocBLAS are crashing nodes, seemingly due to the excessive number of threads and/or due to memory usage. Limiting this to 8, and making sure that Tensile also only uses 8 threads (see easybuilders/easybuild-easyconfigs#26846, something similar was already done for hipBLASLt: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/h/hipBLASLt/hipBLASLt-0.12.1-rocm-compilers-19.0.0-ROCm-6.4.1.eb#L66C1-L66C52) solves the issues.

Do note that these builds require a lot of memory (>50-60GB or even more) for certain GFX targets (in particular gfx942), so they have to be done on larger build nodes. For this we temporarily switched the bot configs to one that contains larger build nodes (-xl nodes), restarted the event handler, submitted the jobs, and reverted back to the original config.

boegel
boegel previously approved these changes Sep 4, 2026

@boegel boegel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel

boegel commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 4, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2023.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15592854

date job status comment
Sep 04 17:39:10 UTC 2026 submitted job id 15592854 awaits release by job manager
Sep 04 17:40:01 UTC 2026 released job awaits launch by Slurm scheduler
Sep 04 17:42:14 UTC 2026 running job 15592854 is running
Sep 04 17:44:25 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15592854.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-17885437940.tar.gzsize: 0 MiB (31526 bytes)
entries: 1
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2023.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2023.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2023.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 04 17:44:25 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15592854.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 4, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2025.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15592855

date job status comment
Sep 04 17:39:16 UTC 2026 submitted job id 15592855 awaits release by job manager
Sep 04 17:39:57 UTC 2026 released job awaits launch by Slurm scheduler
Sep 04 17:41:05 UTC 2026 running job 15592855 is running
Sep 04 17:42:16 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15592855.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-nvidia-grace-17885436700.tar.gzsize: 0 MiB (31527 bytes)
entries: 1
modules under 2025.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 04 17:42:16 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15592855.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 4, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2026.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15592856

date job status comment
Sep 04 17:39:22 UTC 2026 submitted job id 15592856 awaits release by job manager
Sep 04 17:39:53 UTC 2026 released job awaits launch by Slurm scheduler
Sep 04 17:41:10 UTC 2026 running job 15592856 is running
Sep 04 17:42:19 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15592856.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2026.06-software-linux-aarch64-nvidia-grace-17885436490.tar.gzsize: 0 MiB (31528 bytes)
entries: 1
modules under 2026.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2026.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2026.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2026.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 04 17:42:19 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15592856.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@bedroge
bedroge marked this pull request as draft September 4, 2026 17:41
@bedroge

bedroge commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Still testing this (using the commit from this PR) in EESSI/software-layer#1549.

@boegel

boegel commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Still testing this (using the commit from this PR) in EESSI/software-layer#1549.

OK, let us know if it works as expected/hoped...

@ocaisa

ocaisa commented Sep 14, 2026

Copy link
Copy Markdown
Member

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 14, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2023.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15618744

date job status comment
Sep 14 07:17:22 UTC 2026 submitted job id 15618744 awaits release by job manager
Sep 14 07:17:54 UTC 2026 released job awaits launch by Slurm scheduler
Sep 14 08:06:09 UTC 2026 running job 15618744 is running
Sep 14 08:09:28 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15618744.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-17893732640.tar.gzsize: 0 MiB (31533 bytes)
entries: 1
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2023.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2023.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2023.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 14 08:09:28 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15618744.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Sep 15 11:56:04 UTC 2026 uploaded transfer of eessi-2023.06-software-linux-aarch64-nvidia-grace-17893732640.tar.gz to S3 bucket succeeded

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 14, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2025.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15618747

date job status comment
Sep 14 07:17:28 UTC 2026 submitted job id 15618747 awaits release by job manager
Sep 14 07:17:51 UTC 2026 released job awaits launch by Slurm scheduler
Sep 14 08:06:05 UTC 2026 running job 15618747 is running
Sep 14 08:09:24 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15618747.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-nvidia-grace-17893732510.tar.gzsize: 0 MiB (31535 bytes)
entries: 1
modules under 2025.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 14 08:09:24 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15618747.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Sep 15 11:56:14 UTC 2026 uploaded transfer of eessi-2025.06-software-linux-aarch64-nvidia-grace-17893732510.tar.gz to S3 bucket succeeded

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 14, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2026.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15618751

date job status comment
Sep 14 07:17:33 UTC 2026 submitted job id 15618751 awaits release by job manager
Sep 14 07:17:47 UTC 2026 released job awaits launch by Slurm scheduler
Sep 14 08:09:22 UTC 2026 running job 15618751 is running
Sep 14 08:11:33 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15618751.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2026.06-software-linux-aarch64-nvidia-grace-17893733660.tar.gzsize: 0 MiB (31536 bytes)
entries: 1
modules under 2026.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2026.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2026.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2026.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 14 08:11:33 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15618751.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Sep 15 11:55:52 UTC 2026 uploaded transfer of eessi-2026.06-software-linux-aarch64-nvidia-grace-17893733660.tar.gz to S3 bucket succeeded

@bedroge bedroge changed the title use 4 cores for building hipBLASLt use 8 cores for building hipBLASLt and rocBLAS Sep 14, 2026
@bedroge bedroge added the AMD GPU Related to AMD GPU support label Sep 14, 2026
@bedroge
bedroge marked this pull request as ready for review September 14, 2026 07:24
@boegel

boegel commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

@bedroge Can you update the PR description a bit to explain why this is necessary?

There was a lot of back & forth, but it the end it's a pretty clean change.

It does still require xl nodes though due to excessive memory requirements?

@bedroge

bedroge commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@bedroge Can you update the PR description a bit to explain why this is necessary?

There was a lot of back & forth, but it the end it's a pretty clean change.

It does still require xl nodes though due to excessive memory requirements?

Done!

@bedroge

bedroge commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@ocaisa / @boegel ready to deploy.

@boegel boegel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ocaisa

ocaisa commented Sep 15, 2026

Copy link
Copy Markdown
Member

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 15, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2023.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15623810

date job status comment
Sep 15 11:55:20 UTC 2026 submitted job id 15623810 awaits release by job manager
Sep 15 11:55:29 UTC 2026 released job awaits launch by Slurm scheduler
Sep 15 15:48:51 UTC 2026 running job 15623810 is running
Sep 15 15:54:32 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15623810.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-aarch64-nvidia-grace-17894876020.tar.gzsize: 0 MiB (31533 bytes)
entries: 1
modules under 2023.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2023.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2023.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2023.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 15 15:54:32 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15623810.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 15, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2025.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15623811

date job status comment
Sep 15 11:55:30 UTC 2026 submitted job id 15623811 awaits release by job manager
Sep 15 11:56:40 UTC 2026 released job awaits launch by Slurm scheduler
Sep 15 15:48:55 UTC 2026 running job 15623811 is running
Sep 15 15:51:12 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15623811.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-nvidia-grace-17894874020.tar.gzsize: 0 MiB (31534 bytes)
entries: 1
modules under 2025.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 15 15:51:12 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15623811.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@eessi-bot-jsc

eessi-bot-jsc Bot commented Sep 15, 2026

Copy link
Copy Markdown

New job on instance eessi-bot-jsc for repository eessi.io-2026.06-software
Building on: nvidia-grace
Building for: aarch64/nvidia/grace
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.09/pr_302/15623812

date job status comment
Sep 15 11:55:38 UTC 2026 submitted job id 15623812 awaits release by job manager
Sep 15 11:56:35 UTC 2026 released job awaits launch by Slurm scheduler
Sep 15 15:51:07 UTC 2026 running job 15623812 is running
Sep 15 15:58:55 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15623812.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2026.06-software-linux-aarch64-nvidia-grace-17894876730.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2026.06/software/linux/aarch64/nvidia/grace/modules/all
no module files in tarball
software under 2026.06/software/linux/aarch64/nvidia/grace/software
no software packages in tarball
reprod directories under 2026.06/software/linux/aarch64/nvidia/grace/reprod
no reprod directories in tarball
other under 2026.06/software/linux/aarch64/nvidia/grace
no other files in tarball
Sep 15 15:58:55 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15623812.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@boegel

boegel commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@ocaisa Why the rebuild?

@ocaisa

ocaisa commented Sep 15, 2026

Copy link
Copy Markdown
Member

Sorry, I thought the updated comment was in the easystack itself, didn't check properly if there was a commit

@boegel

boegel commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

bot:cancel jobid:15623810

@boegel

boegel commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

bot:cancel jobid:15623811

@boegel

boegel commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

bot:cancel jobid:15623812

@boegel

boegel commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

staging PR merged

@boegel
boegel merged commit 180782b into EESSI:main Sep 15, 2026
94 of 96 checks passed
@bedroge
bedroge deleted the hipblast_max_cores branch September 15, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AMD GPU Related to AMD GPU support bot:deploy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants