-
Notifications
You must be signed in to change notification settings - Fork 405
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failing Kokkos::HIP Unit Tests #6968
Comments
How do these unit tests fail for you? What do they print to the console if you run them individually? |
@masterleinad @Rombur any debug steps I should take? |
@pvelesko It's not obvious what's the problem and I don't have access to these old GPUs anymore. You could try an older version of Kokkos (maybe 4.0) and see if that helps. |
Using HIPAMD 5.6.0 and Kokkos 4.0.00. I have a dependency on Kokkos 4.3.00
|
What is the actual GPU you are trying to use? I am running on an MI50 and it seems all to work with your command. |
(Note I was using Kokkos 4.3.01 and ROCM 6.0 not 6.1 which I didn't have on my machine) |
Output from incremental test:
|
Note I have a few test failures for the math special functions which don't produce the results within the tolerances we expect, and in some cases NAN and INF seem to be handled differently on MI50 than on MI200, but none of the failures you are reporting. |
I am wondering if it's a problem with the driver. I.e. that a newer driver than ROCM 6.0 doesn't work anymore for MI50. |
Just looked it up support for MI50 is deprecated: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html I wonder how well they test the drivers, and your specific failure modes (with stuff like pages not mapped) could be an indication of a driver issue. |
Vega 20 which gfx906 or an MI consumer grade equivalent. So it's deprecated but still supported.
I can downgrade the driver and test. |
We tried to reach out to you on slack to discuss this pull request and #7007. Do you have an active handle there? |
Ah I was signed out of Kokkos workspace. I'll check |
Using Kokkos:: tag 4.3.00 the following unit tests fail
Please include the following for a minimal reproducer
run tests
ctest
KokkosCore_config.h
header file (generated during the build)The text was updated successfully, but these errors were encountered: