Sanity Checking DataLoader error

[DieTANG]

Hi

I am running the quick start command to check my hyena-dna installation.
python -m train wandb=null experiment=hg38/genomic_benchmark_scratch
and I got the error report below, does anyone know how to fix this?

Sanity Checking DataLoader 0:   0%|                                                                      | 0/2 [00:00<?, ?it/s]
CUDA Error: invalid device function /PATH/hyena-dna/flash-attention/csrc/layer_norm/ln_fwd_kernels.cuh 271

Best
Dié

[TaichiMatsubara]

Hi

I receive same error at same execution

Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]
Segmentation fault

It's a segmentation fault error.

[countrsignal]

I'm running into this exact problem as well. Any solutions?

[countrsignal]

Ok I was able to resolve this issue for my setup. The Issue for me was that the CUDA drivers for my GPU's were incompatible with the flash_attn build. Make sure you have pytorch with cuda/11.7 installed. When I re-installed flash_attn, I used gcc/11.3.0 and cuda/11.7.1.

If you encounter this problem, try installing pytorch/2.0.1 with cuda/11.7. Hope this is somewhat helpful.