Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cuda graph capture may occasionally become stuck with multiple gpus. #131

Open
guocuimi opened this issue Apr 18, 2024 · 0 comments
Open

cuda graph capture may occasionally become stuck with multiple gpus. #131

guocuimi opened this issue Apr 18, 2024 · 0 comments
Assignees
@liutongxuan
Labels
bug Something isn't working enhancement New feature or request

Comments

@guocuimi
Copy link
Collaborator

It is a known issue that CUDA graph capture may occasionally become stuck when multiple workers are in use. further investigation is needed.

Disabled cuda graph for multiple gpus by default for now.
@guocuimi guocuimi mentioned this issue Apr 18, 2024
Merged
@guocuimi guocuimi self-assigned this Apr 18, 2024
guocuimi added a commit that referenced this issue Apr 18, 2024
@guocuimi
[bugfix] fix cuda graph capture issue for tensor parallelism (#130)
10f590d 
It is a known issue (#131)
that CUDA graph capture may occasionally become stuck with multiple
gpus.
Disable cuda graph for multiple gpus for now and investigate further
later.
@guocuimi guocuimi assigned liutongxuan and unassigned guocuimi Apr 19, 2024
@guocuimi guocuimi added bug Something isn't working enhancement New feature or request labels Apr 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liutongxuan liutongxuan

Labels
bug Something isn't working enhancement New feature or request
Projects
ScaleLLM 2024 H1 Roadmap
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@liutongxuan @guocuimi