Skip to content

feat: add CPU staging buffer to access GPU memory via unified buffer - #37

Open
WillowWang0216 wants to merge 3 commits into
LinQuickDev:supercachefrom
WillowWang0216:supercache-ub
Open

feat: add CPU staging buffer to access GPU memory via unified buffer#37
WillowWang0216 wants to merge 3 commits into
LinQuickDev:supercachefrom
WillowWang0216:supercache-ub

Conversation

@WillowWang0216

Copy link
Copy Markdown

When integrating vllm with mooncake, GPU memory addresses need to be passed to mooncake for data transfer. However, the existing ubtransport implementation does not support registering GPU memory directly, making it impossible to fetch data from the GPU side.
To address this, a CPU staging buffer mechanism is introduced:
Add a CPU staging area as an intermediate buffer between GPU memory and ubtransport
Before data transfer, copy data from GPU memory to the CPU staging buffer (D2H)
mooncake reads data from the CPU staging buffer via ubtransport
For reverse transfers, data is first written to the CPU staging buffer, then copied back to GPU memory (H2D)
This approach enables indirect access and transfer of GPU memory data without modifying the ubtransport registration logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant