Skip to content

Add optional CUDA workflow on a simulated H100 - #687

Open
saqibkh wants to merge 1 commit into
ParRes:mainfrom
saqibkh:add-gpu-sim-ci
Open

saqibkh wants to merge 1 commit into
ParRes:mainfrom
saqibkh:add-gpu-sim-ci

Conversation

@saqibkh

@saqibkh saqibkh commented Nov 9, 2026

Copy link
Copy Markdown

The CI has no job that builds or runs the CUDA kernels in Cxx11/, because the hosted runners have no GPU. This adds a separate, optional workflow (.github/workflows/gpu-sim.yml) that builds them with CUDA 13.0 and runs them on a simulated NVIDIA H100 provided by PantheonSim (via the pantheongpu/setup-pantheonsim action, pinned by SHA). I am the author of PantheonSim, so please weigh this accordingly.

It is not meant to be a required check and does not replace testing on real GPUs. A simulator cannot reproduce hardware-specific behavior or timing, so I see it as a way to catch CUDA breakage in pull requests when nobody has a GPU at hand. The programs verify their own results, so a wrong answer makes the step fail.

What it does

  • Copies common/make.defs.cuda to common/make.defs and changes three things with sed: NVCC points at the CUDA 13.0 toolkit installed by the action, --gpu-architecture=sm_89 becomes compute_90 -cudart shared (the simulator JIT-compiles PTX and needs a shared CUDA runtime), and the CUDA 12.6 include path is updated. Nothing in the repository's own files is changed.
  • Builds with make -f Makefile.legacy: nstream-cuda, nstream-managed-cuda, transpose-cuda, stencil-cuda, p2p-cuda, nstream-device-thrust, transpose-device-thrust, nstream-cublas, transpose-cublas, dgemm-cublas, sgemm-cublas. Boost headers are installed because the thrust and cuBLAS targets include Boost.Range.
  • Runs each program under vgpu run with small sizes (19 invocations: a few variants of nstream, transpose, stencil, p2p, and one or two of the others). There is no CUDA entry in the existing test scripts, so the arguments follow each program's usage message.
  • Triggers: workflow_dispatch and pull_request to main, path-filtered to the CUDA, cuBLAS and thrust sources, prk_*.h, Cxx11/Makefile.legacy, common/make.defs.cuda and the workflow itself. A concurrency group cancels older runs.

What was left out

  • The *-cudastf programs (need the CUDA STF headers from ci/install-cudastf.sh), and the MPI, NCCL, NVSHMEM and multi-GPU programs.
  • Only one GPU (an H100) and one CUDA version (13.0) are covered.

Result on a fork

All 19 runs print "Solution validates" and the job succeeds. With a cold cache the job took about 16 minutes, almost all of it the action building the simulator the first time; building the kernels took 28 s and running them took 14 s.

Run: https://github.com/saqibkh/Kernels/actions/runs/42557389354

@saqibkh
saqibkh requested a review from jeffhammond as a code owner November 9, 2026 03:54
@saqibkh

saqibkh commented Nov 13, 2026

Copy link
Copy Markdown
Author

The red allcxx/allfortran/alljulia jobs aren't from this PR, which only adds gpu-sim.yml. The macOS ones also fail on main. The ubuntu allcxx jobs fail because ubuntu-latest is now 26.04, where ci/install-gcc.sh can't find any of gcc-10 down to gcc-5. The simulated H100 job passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants