Jitify is performing a one-time only warm-up to populate the persistent cache

[ryanshrott]

Description

Get's suck here forever! "Jitify is performing a one-time only warm-up to populate the persistent cache, this may take a few seconds and will be improved in a future release...
jitify._init_module()"

`(rapids_new) (base) azureuser@ryanshrott1:~/cloudfiles/code/propertyca_space$ python
Python 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import dask_cudf
import cudf
import cupy as cp
s = cudf.Series([1, 2, 3, None, 4])
ds = dask_cudf.from_cudf(s, npartitions=2)
/anaconda/envs/rapids_new/lib/python3.11/site-packages/cupy/cuda/compiler.py:233: PerformanceWarning: Jitify is performing a one-time only warm-up to populate the persistent cache, this may take a few seconds and will be improved in a future release...
jitify._init_module()`

To Reproduce

>>> import dask_cudf
>>> import cudf
>>> import cupy as cp
>>> s = cudf.Series([1, 2, 3, None, 4])
>>> ds = dask_cudf.from_cudf(s, npartitions=2)
/anaconda/envs/rapids_new/lib/python3.11/site-packages/cupy/cuda/compiler.py:233: PerformanceWarning: Jitify is performing a one-time only warm-up to populate the persistent cache, this may take a few seconds and will be improved in a future release...
  jitify._init_module()``````


### Installation
https://docs.rapids.ai/install

conda create -n rapids-24.04 -c rapidsai -c conda-forge -c nvidia  \
    rapids=24.04 python=3.11 cuda-version=12.2 \
    pytorch
### Environment

(rapids_new) (base) azureuser@ryanshrott1:~/cloudfiles/code/propertyca_space$ python
Python 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import dask_cudf
import cudf
import cupy as cp
s = cudf.Series([1, 2, 3, None, 4])
ds = dask_cudf.from_cudf(s, npartitions=2)
/anaconda/envs/rapids_new/lib/python3.11/site-packages/cupy/cuda/compiler.py:233: PerformanceWarning: Jitify is performing a one-time only warm-up to populate the persistent cache, this may take a few seconds and will be improved in a future release...
jitify._init_module()```

Additional Information

No response

[leofang]

Could you please fill in all the issue template? It's unclear how you installed CuPy or what your environment is.

[ryanshrott]

Sure, just updated it

[leofang]

Could you share the outputs of the following?

• python -c "import cupy; cupy.show_config()"
• conda list
• conda info

[ryanshrott]

`(rapids_new) (base) azureuser@ryanshrott1:/cloudfiles/code$ python -c "import cupy; cupy.show_config()"
OS : Linux-5.15.0-1040-azure-x86_64-with-glibc2.31
Python Version : 3.11.9
CuPy Version : 13.1.0
CuPy Platform : NVIDIA CUDA
NumPy Version : 1.26.4
SciPy Version : 1.13.0
Cython Build Version : 0.29.37
Cython Runtime Version : None
CUDA Root : /anaconda/envs/rapids_new
nvcc PATH : /anaconda/envs/rapids_new/bin/nvcc
CUDA Build Version : 12040
CUDA Driver Version : 12020
CUDA Runtime Version : 12040 (linked to CuPy) / 12020 (locally installed)
cuBLAS Version : (available)
cuFFT Version : 11008
cuRAND Version : 10303
cuSOLVER Version : (11, 5, 2)
cuSPARSE Version : (available)
NVRTC Version : (12, 2)
Thrust Version : 200302
CUB Build Version : 200200
Jitify Build Version :
cuDNN Build Version : 8907
cuDNN Version : 8907
NCCL Build Version : 22105
NCCL Runtime Version : 22105
cuTENSOR Version : None
cuSPARSELt Build Version : None
Device 0 Name : NVIDIA A100 80GB PCIe
Device 0 Compute Capability : 80
Device 0 PCI Bus ID : 0001:00:00.0
(rapids_new) (base) azureuser@ryanshrott1:/cloudfiles/code$ conda list

packages in environment at /anaconda/envs/rapids_new:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
aiohttp 3.9.5 py311h459d7ec_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 4.3.0 pyhd8ed1ab_0 conda-forge
aom 3.8.2 h59595ed_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py311h459d7ec_4 conda-forge
arrow 1.3.0 pyhd8ed1ab_0 conda-forge
attrs 23.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.7.16 h79b3bcb_6 conda-forge
aws-c-cal 0.6.10 hb29e0c7_1 conda-forge
aws-c-common 0.9.13 hd590300_0 conda-forge
aws-c-compression 0.2.18 hecc5fa9_1 conda-forge
aws-c-event-stream 0.4.2 hf9b2f7b_4 conda-forge
aws-c-http 0.8.1 h5d7533a_5 conda-forge
aws-c-io 0.14.5 h50678d4_1 conda-forge
aws-c-mqtt 0.10.2 hf479d2b_4 conda-forge
aws-c-s3 0.5.2 h4ad9680_0 conda-forge
aws-c-sdkutils 0.1.15 hecc5fa9_1 conda-forge
aws-checksums 0.1.18 hecc5fa9_1 conda-forge
aws-crt-cpp 0.26.2 h19f5d62_7 conda-forge
aws-sdk-cpp 1.11.267 h5606698_1 conda-forge
azure-core-cpp 1.11.1 h91d86a7_1 conda-forge
azure-storage-blobs-cpp 12.10.0 h00ab1b0_1 conda-forge
azure-storage-common-cpp 12.5.0 h94269e2_4 conda-forge
beautifulsoup4 4.12.3 pyha770c72_0 conda-forge
bleach 6.1.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.5 h0f2a231_0 conda-forge
bokeh 3.4.1 pyhd8ed1ab_0 conda-forge
branca 0.7.2 pyhd8ed1ab_0 conda-forge
brotli 1.1.0 hd590300_1 conda-forge
brotli-bin 1.1.0 hd590300_1 conda-forge
brotli-python 1.1.0 py311hb755f60_1 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.28.1 hd590300_0 conda-forge
c-blosc2 2.14.3 hb4ffafa_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.3.3 pyhd8ed1ab_0 conda-forge
cairo 1.18.0 h3faef2a_0 conda-forge
catboost 1.2.5 pypi_0 pypi
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py311hb3a22ac_0 conda-forge
cfitsio 4.3.1 hbdc6101_0 conda-forge
charls 2.4.2 h59595ed_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
clickhouse-connect 0.7.8 pypi_0 pypi
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
colorcet 3.1.0 pyhd8ed1ab_0 conda-forge
contourpy 1.2.1 py311h9547e67_0 conda-forge
cucim 24.04.00 cuda12_py311_240410_ga24abfd_0 rapidsai
cuda-cccl_linux-64 12.2.140 ha770c72_0 conda-forge
cuda-crt-dev_linux-64 12.2.140 ha770c72_1 conda-forge
cuda-crt-tools 12.2.140 ha770c72_1 conda-forge
cuda-cudart 12.2.140 hd3aeb46_0 conda-forge
cuda-cudart-dev 12.2.140 hd3aeb46_0 conda-forge
cuda-cudart-dev_linux-64 12.2.140 h59595ed_0 conda-forge
cuda-cudart-static 12.2.140 hd3aeb46_0 conda-forge
cuda-cudart-static_linux-64 12.2.140 h59595ed_0 conda-forge
cuda-cudart_linux-64 12.2.140 h59595ed_0 conda-forge
cuda-nvcc-dev_linux-64 12.2.140 ha770c72_1 conda-forge
cuda-nvcc-impl 12.2.140 hd3aeb46_1 conda-forge
cuda-nvcc-tools 12.2.140 hd3aeb46_1 conda-forge
cuda-nvrtc 12.2.140 hd3aeb46_0 conda-forge
cuda-nvtx 12.2.140 h59595ed_0 conda-forge
cuda-nvvm-dev_linux-64 12.2.140 ha770c72_1 conda-forge
cuda-nvvm-impl 12.2.140 h59595ed_1 conda-forge
cuda-nvvm-tools 12.2.140 h59595ed_1 conda-forge
cuda-profiler-api 12.2.140 ha770c72_0 conda-forge
cuda-python 12.4.0 py311h817de4b_2 conda-forge
cuda-version 12.2 he2b69de_3 conda-forge
cudf 24.04.01 cuda12_py311_240501_gaf9fd840f2_0 rapidsai
cudf_kafka 24.04.01 cuda12_py311_240501_gaf9fd840f2_0 rapidsai
cudnn 8.9.7.29 h092f7fd_3 conda-forge
cugraph 24.04.00 cuda12_py311_240410_g8d8fb74c2_0 rapidsai
cuml 24.04.00 cuda12_py311_240410_ge7bc67431_0 rapidsai
cuproj 24.04.00 cuda12_py311_240410_g7da37aae_0 rapidsai
cupy 13.1.0 py311hf829483_4 conda-forge
cupy-core 13.1.0 py311he1e6e68_4 conda-forge
cuspatial 24.04.00 cuda12_py311_240410_g7da37aae_0 rapidsai
custreamz 24.04.01 cuda12_py311_240501_gaf9fd840f2_0 rapidsai
cuxfilter 24.04.01 cuda12_py311_240411_g175a4dc_0 rapidsai
cycler 0.12.1 pyhd8ed1ab_0 conda-forge
cyrus-sasl 2.1.27 h54b06d7_7 conda-forge
cytoolz 0.12.3 py311h459d7ec_0 conda-forge
dask 2024.1.1 pyhd8ed1ab_0 conda-forge
dask-core 2024.1.1 pyhd8ed1ab_0 conda-forge
dask-cuda 24.04.00 py311_240410_g7ed6735_0 rapidsai
dask-cudf 24.04.01 cuda12_py311_240501_gaf9fd840f2_0 rapidsai
dask-expr 0.4.0 pyhd8ed1ab_0 conda-forge
datasets 2.19.1 pypi_0 pypi
datashader 0.16.1 pyhd8ed1ab_0 conda-forge
dav1d 1.2.1 hd590300_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
dill 0.3.8 pypi_0 pypi
distributed 2024.1.1 pyhd8ed1ab_0 conda-forge
dlpack 0.8 h59595ed_3 conda-forge
duckdb 0.10.2 pypi_0 pypi
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge
expat 2.6.2 h59595ed_0 conda-forge
fastrlock 0.8.2 py311hb755f60_2 conda-forge
filelock 3.14.0 pyhd8ed1ab_0 conda-forge
fiona 1.9.5 py311hf8e0aa6_2 conda-forge
fmt 10.2.1 h00ab1b0_0 conda-forge
folium 0.16.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_2 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.51.0 py311h459d7ec_0 conda-forge
fqdn 1.5.1 pyhd8ed1ab_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
freexl 2.0.0 h743c826_0 conda-forge
frozenlist 1.4.1 py311h459d7ec_0 conda-forge
fsspec 2024.3.1 pyhca7485f_0 conda-forge
gdal 3.8.1 py311h39b4e0e_3 conda-forge
geopandas 0.14.4 pyhd8ed1ab_0 conda-forge
geopandas-base 0.14.4 pyha770c72_0 conda-forge
geos 3.12.1 h59595ed_0 conda-forge
geotiff 1.7.1 hf074850_14 conda-forge
gettext 0.22.5 h59595ed_2 conda-forge
gettext-tools 0.22.5 h59595ed_2 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.2 hd590300_0 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
gmock 1.14.0 ha770c72_1 conda-forge
gmp 6.3.0 h59595ed_1 conda-forge
gmpy2 2.1.5 py311he48d604_0 conda-forge
gtest 1.14.0 h00ab1b0_1 conda-forge
hdf4 4.2.15 h2a13503_7 conda-forge
hdf5 1.14.3 nompi_h4f84152_101 conda-forge
holoviews 1.18.3 pyhd8ed1ab_0 conda-forge
huggingface-hub 0.23.0 pypi_0 pypi
icu 73.2 h59595ed_0 conda-forge
idna 3.7 pyhd8ed1ab_0 conda-forge
imagecodecs 2024.1.1 py311hd0e15ba_2 conda-forge
imageio 2.34.1 pyh4b66e23_0 conda-forge
importlib-metadata 7.1.0 pyha770c72_0 conda-forge
importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge
importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge
isoduration 20.11.0 pyhd8ed1ab_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
joblib 1.4.2 pyhd8ed1ab_0 conda-forge
json-c 0.17 h7ab15ed_0 conda-forge
jsonpointer 2.4 py311h38be061_3 conda-forge
jsonschema 4.22.0 pyhd8ed1ab_0 conda-forge
jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge
jsonschema-with-format-nongpl 4.22.0 pyhd8ed1ab_0 conda-forge
jupyter-server-proxy 4.1.2 pyhd8ed1ab_0 conda-forge
jupyter_client 8.6.1 pyhd8ed1ab_0 conda-forge
jupyter_core 5.7.2 py311h38be061_0 conda-forge
jupyter_events 0.10.0 pyhd8ed1ab_0 conda-forge
jupyter_server 2.14.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.5.3 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.3.0 pyhd8ed1ab_1 conda-forge
jxrlib 1.1 hd590300_3 conda-forge
kealib 1.5.3 h2f55d51_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.5 py311h9547e67_1 conda-forge
krb5 1.21.2 h659d440_0 conda-forge
lazy_loader 0.4 pyhd8ed1ab_0 conda-forge
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.40 h55db66e_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230802.1 cxx17_h59595ed_0 conda-forge
libaec 1.1.3 h59595ed_0 conda-forge
libarchive 3.7.2 h2aa1ff5_1 conda-forge
libarrow 14.0.2 h49c8883_7_cpu conda-forge
libarrow-acero 14.0.2 h59595ed_7_cpu conda-forge
libarrow-dataset 14.0.2 h59595ed_7_cpu conda-forge
libarrow-flight 14.0.2 hdc44a87_7_cpu conda-forge
libarrow-flight-sql 14.0.2 hfbc7f12_7_cpu conda-forge
libarrow-gandiva 14.0.2 h308e607_7_cpu conda-forge
libarrow-substrait 14.0.2 hfbc7f12_7_cpu conda-forge
libasprintf 0.22.5 h661eb56_2 conda-forge
libasprintf-devel 0.22.5 h661eb56_2 conda-forge
libavif16 1.0.4 hd9d6309_2 conda-forge
libblas 3.9.0 22_linux64_openblas conda-forge
libboost-headers 1.84.0 ha770c72_2 conda-forge
libbrotlicommon 1.1.0 hd590300_1 conda-forge
libbrotlidec 1.1.0 hd590300_1 conda-forge
libbrotlienc 1.1.0 hd590300_1 conda-forge
libcblas 3.9.0 22_linux64_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcublas 12.2.5.6 hd3aeb46_0 conda-forge
libcublas-dev 12.2.5.6 hd3aeb46_0 conda-forge
libcucim 24.04.00 cuda12_240410_ga24abfd_0 rapidsai
libcudf 24.04.01 cuda12_240501_gaf9fd840f2_0 rapidsai
libcudf_kafka 24.04.01 cuda12_240501_gaf9fd840f2_0 rapidsai
libcufft 11.0.8.103 hd3aeb46_0 conda-forge
libcufile 1.7.2.10 hd3aeb46_0 conda-forge
libcufile-dev 1.7.2.10 hd3aeb46_0 conda-forge
libcugraph 24.04.00 cuda12_240410_g8d8fb74c2_0 rapidsai
libcugraph_etl 24.04.00 cuda12_240410_g8d8fb74c2_0 rapidsai
libcugraphops 24.04.00 cuda12_240410_g021aa305_0 nvidia
libcuml 24.04.00 cuda12_240410_ge7bc67431_0 rapidsai
libcumlprims 24.04.00 cuda12_240410_gfae5daf_0 nvidia
libcurand 10.3.3.141 hd3aeb46_0 conda-forge
libcurand-dev 10.3.3.141 hd3aeb46_0 conda-forge
libcurl 8.7.1 hca28451_0 conda-forge
libcusolver 11.5.2.141 hd3aeb46_0 conda-forge
libcusolver-dev 11.5.2.141 hd3aeb46_0 conda-forge
libcusparse 12.1.2.141 hd3aeb46_0 conda-forge
libcusparse-dev 12.1.2.141 hd3aeb46_0 conda-forge
libcuspatial 24.04.00 cuda12_240410_g7da37aae_0 rapidsai
libdeflate 1.19 hd590300_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.12 hf998b51_1 conda-forge
libexpat 2.6.2 h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_7 conda-forge
libgdal 3.8.1 h4b8bffa_3 conda-forge
libgettextpo 0.22.5 h59595ed_2 conda-forge
libgettextpo-devel 0.22.5 h59595ed_2 conda-forge
libgfortran-ng 13.2.0 h69a702a_7 conda-forge
libgfortran5 13.2.0 hca663fb_7 conda-forge
libglib 2.78.4 h783c2da_0 conda-forge
libgoogle-cloud 2.12.0 hef10d8f_5 conda-forge
libgrpc 1.60.1 h74775cd_0 conda-forge
libhwloc 2.10.0 default_h2fb2949_1000 conda-forge
libhwy 1.1.0 h00ab1b0_0 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
libjxl 0.10.2 hcae5a98_0 conda-forge
libkml 1.3.0 h01aab08_1018 conda-forge
libkvikio 24.04.00 cuda12_240410_gfb767d7_0 rapidsai
liblapack 3.9.0 22_linux64_openblas conda-forge
libllvm14 14.0.6 hcd5def8_4 conda-forge
libllvm15 15.0.7 hb3ce162_4 conda-forge
libmagma 2.7.2 h173bb3b_2 conda-forge
libmagma_sparse 2.7.2 h173bb3b_3 conda-forge
libnetcdf 4.9.2 nompi_h9612171_113 conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnl 3.9.0 hd590300_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libntlm 1.4 h7f98852_1002 conda-forge
libnvjitlink 12.2.140 hd3aeb46_0 conda-forge
libnvjpeg 12.2.2.4 h59595ed_0 conda-forge
libopenblas 0.3.27 pthreads_h413a1c8_0 conda-forge
libparquet 14.0.2 h352af49_7_cpu conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libpq 16.2 h33b98f1_1 conda-forge
libprotobuf 4.25.1 hf27288f_2 conda-forge
libraft 24.04.00 cuda12_240410_gfa44bcc8_0 rapidsai
libraft-headers 24.04.00 cuda12_240410_gfa44bcc8_0 rapidsai
libraft-headers-only 24.04.00 cuda12_240410_gfa44bcc8_0 rapidsai
librdkafka 1.9.2 ha5a0de0_2 conda-forge
libre2-11 2023.09.01 h7a70373_1 conda-forge
librmm 24.04.00 cuda12_240410_g8f19c9c3_0 rapidsai
librttopo 1.1.0 h8917695_15 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.1.0 h72606ae_3 conda-forge
libsqlite 3.45.3 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge
libthrift 0.19.0 hb90f79a_1 conda-forge
libtiff 4.6.0 ha9c0a0a_2 conda-forge
libtorch 2.1.2 cuda120_h2aa5df7_303 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.48.0 hd590300_0 conda-forge
libwebp 1.3.2 h658648e_1 conda-forge
libwebp-base 1.3.2 hd590300_1 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxgboost 2.0.3 rapidsai_hea4bf7d_2 rapidsai
libxml2 2.12.6 h232c23b_2 conda-forge
libzip 1.10.1 h2629f0a_3 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
linkify-it-py 2.0.3 pyhd8ed1ab_0 conda-forge
llvm-openmp 18.1.5 ha31de31_0 conda-forge
llvmlite 0.42.0 py311ha6695c7_1 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.3 py311h38e4bf4_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 hd590300_1001 conda-forge
magma 2.7.2 h51420fd_3 conda-forge
mapclassify 2.6.1 pyhd8ed1ab_0 conda-forge
markdown 3.6 pyhd8ed1ab_0 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py311h459d7ec_0 conda-forge
matplotlib-base 3.8.4 py311h54ef318_0 conda-forge
mdit-py-plugins 0.4.0 pyhd8ed1ab_0 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
minizip 4.0.5 h0ab5242_0 conda-forge
mistune 3.0.2 pyhd8ed1ab_0 conda-forge
mkl 2023.2.0 h84fe81f_50496 conda-forge
mpc 1.3.1 hfe3b2da_0 conda-forge
mpfr 4.2.1 h9458935_1 conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.7 py311h9547e67_0 conda-forge
multidict 6.0.5 py311h459d7ec_0 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
multiprocess 0.70.16 pypi_0 pypi
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nbclient 0.10.0 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.16.4 pyhd8ed1ab_0 conda-forge
nbformat 5.10.4 pyhd8ed1ab_0 conda-forge
nccl 2.21.5.1 h3a97aeb_0 conda-forge
ncurses 6.4.20240210 h59595ed_0 conda-forge
networkx 3.3 pyhd8ed1ab_1 conda-forge
nodejs 20.12.2 hb753e55_0 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.98 h1d7d5a4_0 conda-forge
numba 0.59.1 py311h96b013e_0 conda-forge
numpy 1.26.4 py311h64a7726_0 conda-forge
nvcomp 3.0.6 h10b603f_0 conda-forge
nvtx 0.2.10 py311h459d7ec_0 conda-forge
nx-cugraph 24.04.00 py311_0 rapidsai
openjpeg 2.5.2 h488ebb8_0 conda-forge
openssl 3.3.0 hd590300_0 conda-forge
orc 1.9.2 h7829240_1 conda-forge
overrides 7.7.0 pyhd8ed1ab_0 conda-forge
packaging 24.0 pyhd8ed1ab_0 conda-forge
pandas 2.2.1 py311h320fe9a_0 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
panel 1.4.2 pyhd8ed1ab_0 conda-forge
param 2.1.0 pyhca7485f_0 conda-forge
partd 1.4.2 pyhd8ed1ab_0 conda-forge
pcre2 10.42 hcad00b1_0 conda-forge
pillow 10.3.0 py311h18e6fac_0 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
pixman 0.43.2 h59595ed_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
platformdirs 4.2.1 pyhd8ed1ab_0 conda-forge
plotly 5.22.0 pypi_0 pypi
poppler 23.12.0 h590f24d_0 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 16.2 h82ecc9d_1 conda-forge
proj 9.3.0 h1d62c97_2 conda-forge
prometheus_client 0.20.0 pyhd8ed1ab_0 conda-forge
protobuf 4.25.1 py311h46cbc50_0 conda-forge
psutil 5.9.8 py311h459d7ec_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
py-xgboost 2.0.3 rapidsai_pyhf96fb25_2 rapidsai
pyarrow 14.0.2 py311h39c9aba_7_cpu conda-forge
pyarrow-hotfix 0.6 pyhd8ed1ab_0 conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pyct 0.5.0 pyhd8ed1ab_0 conda-forge
pyee 8.1.0 pyhd8ed1ab_0 conda-forge
pygments 2.18.0 pyhd8ed1ab_0 conda-forge
pylibcugraph 24.04.00 cuda12_py311_240410_g8d8fb74c2_0 rapidsai
pylibraft 24.04.00 cuda12_py311_240410_gfa44bcc8_0 rapidsai
pynvjitlink 0.2.3 py311hdaa3023_0 rapidsai
pynvml 11.4.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge
pyppeteer 1.0.2 pyhd8ed1ab_0 conda-forge
pyproj 3.6.1 py311h1facc83_4 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.11.9 hb806964_0_cpython conda-forge
python-confluent-kafka 1.9.2 py311hd4cff14_2 conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-dotenv 1.0.1 pypi_0 pypi
python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge
python-geohash 0.8.5 pypi_0 pypi
python-graphviz 0.20.3 pypi_0 pypi
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge
python_abi 3.11 4_cp311 conda-forge
pytorch 2.1.2 cuda120_py311h25b6552_303 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pyviz_comms 3.0.1 pyhd8ed1ab_0 conda-forge
pywavelets 1.4.1 py311h1f0f07a_1 conda-forge
pyyaml 6.0.1 py311h459d7ec_1 conda-forge
pyzmq 26.0.3 py311h08a0b41_0 conda-forge
raft-dask 24.04.00 cuda12_py311_240410_gfa44bcc8_0 rapidsai
rapids 24.04.00 cuda12_py311_240411_gb301c96_0 rapidsai
rapids-dask-dependency 24.04.01 py_0 rapidsai
rapids-xgboost 24.04.00 cuda12_py311_240411_gb301c96_0 rapidsai
rav1e 0.6.6 he8a937b_2 conda-forge
rdma-core 51.0 hd3aeb46_0 conda-forge
re2 2023.09.01 h7f4b329_1 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.35.1 pyhd8ed1ab_0 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rich 13.7.1 pyhd8ed1ab_0 conda-forge
rmm 24.04.00 cuda12_py311_240410_g8f19c9c3_0 rapidsai
rpds-py 0.18.1 py311h5ecf98a_0 conda-forge
rtree 1.2.0 py311h3bb2b0f_0 conda-forge
s2n 1.4.5 h06160fa_0 conda-forge
scikit-image 0.22.0 py311h320fe9a_2 conda-forge
scikit-learn 1.4.2 py311hc009520_0 conda-forge
scipy 1.13.0 py311h517d4fd_1 conda-forge
send2trash 1.8.3 pyh0d859eb_0 conda-forge
setuptools 69.5.1 pyhd8ed1ab_0 conda-forge
shapely 2.0.4 py311h2032efe_0 conda-forge
simpervisor 1.0.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sleef 3.5.1 h9b69904_2 conda-forge
snappy 1.1.10 hdb0a2a9_1 conda-forge
sniffio 1.3.1 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.5 pyhd8ed1ab_1 conda-forge
spdlog 1.12.0 hd2e6256_2 conda-forge
sqlite 3.45.3 h2c6b66d_0 conda-forge
streamz 0.6.4 pyh6c4a22f_0 conda-forge
svt-av1 2.0.0 h59595ed_0 conda-forge
sympy 1.12 pypyh9d50eac_103 conda-forge
tbb 2021.12.0 h00ab1b0_0 conda-forge
tblib 3.0.0 pyhd8ed1ab_0 conda-forge
tenacity 8.3.0 pypi_0 pypi
terminado 0.18.1 pyh0d859eb_0 conda-forge
threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge
tifffile 2024.5.3 pyhd8ed1ab_0 conda-forge
tiledb 2.18.5 h4386cac_0 conda-forge
tinycss2 1.3.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
toolz 0.12.1 pyhd8ed1ab_0 conda-forge
tornado 6.4 py311h459d7ec_0 conda-forge
tqdm 4.66.4 pyhd8ed1ab_0 conda-forge
traitlets 5.14.3 pyhd8ed1ab_0 conda-forge
treelite 4.1.2 py311he8f9275_2 conda-forge
types-python-dateutil 2.9.0.20240316 pyhd8ed1ab_0 conda-forge
typing-extensions 4.11.0 hd8ed1ab_0 conda-forge
typing_extensions 4.11.0 pyha770c72_0 conda-forge
typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge
tzcode 2024a h3f72095_0 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
uc-micro-py 1.0.3 pyhd8ed1ab_0 conda-forge
ucx 1.15.0 hda83522_8 conda-forge
ucx-proc 1.0.0 gpu rapidsai
ucx-py 0.37.00 py311_240410_gcb8f84c_0 rapidsai
uri-template 1.3.0 pyhd8ed1ab_0 conda-forge
uriparser 0.9.8 hac33072_0 conda-forge
urllib3 1.26.18 pyhd8ed1ab_0 conda-forge
utm 0.7.0 pypi_0 pypi
webcolors 1.13 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 pyhd8ed1ab_2 conda-forge
websocket-client 1.8.0 pyhd8ed1ab_0 conda-forge
websockets 10.4 py311hd4cff14_1 conda-forge
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xarray 2024.3.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.5 hac6953d_0 conda-forge
xgboost 2.0.3 rapidsai_pyhcb9e655_2 rapidsai
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.1.1 hd590300_0 conda-forge
xorg-libsm 1.2.4 h7391055_0 conda-forge
xorg-libx11 1.8.9 h8ee46fc_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.11 hd590300_0 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xxhash 3.4.1 pypi_0 pypi
xyzservices 2024.4.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.4 py311h459d7ec_0 conda-forge
zeromq 4.3.5 h75354e8_3 conda-forge
zfp 1.0.1 h59595ed_0 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zlib-ng 2.0.7 h0b41bf4_0 conda-forge
zstandard 0.22.0 pypi_0 pypi
zstd 1.5.6 ha6fb4c9_0 conda-forge
(rapids_new) (base) azureuser@ryanshrott1:~/cloudfiles/code$ conda info

 active environment : rapids_new
active env location : /anaconda/envs/rapids_new
        shell level : 2
   user config file : /home/azureuser/.condarc

populated config files :
conda version : 23.5.0
conda-build version : not installed
python version : 3.10.11.final.0
virtual packages : __archspec=1=x86_64
__cuda=12.2=0
__glibc=2.31=0
__linux=5.15.0=0
__unix=0=0
base environment : /anaconda (writable)
conda av data dir : /anaconda/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /anaconda/pkgs
/home/azureuser/.conda/pkgs
envs directories : /anaconda/envs
/home/azureuser/.conda/envs
platform : linux-64
user-agent : conda/23.5.0 requests/2.29.0 CPython/3.10.11 Linux/5.15.0-1040-azure ubuntu/20.04.6 glibc/2.31
UID:GID : 1000:1000
netrc file : None
offline mode : False

(rapids_new) (base) azureuser@ryanshrott1:~/cloudfiles/code$ `

[leofang]

Thanks, though I am clueless... is this a local machine that you own, or a cluster, or cloud environment? Any chance you know the storage type (SSD, NFS, ...)?

[ryanshrott]

It’s an azure cloud instance. I think it’s ssd. But not 100% sure. Sent from my iPhoneOn May 9, 2024, at 6:10 PM, Leo Fang ***@***.***> wrote: Thanks, though I am clueless... is this a local machine that you own, or a cluster, or cloud environment? Any chance you know the storage type (SSD, NFS, ...)? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[jakirkham]

Think Leo is on to something

Would suggest taking a closer look at how the drives are backed. Would start with df -h

Also would check permissions. If the default locations we write to we lack permissions on, that could cause a hang

Depending on what we find we may need to change where CuPy caches kernels. Dask may need similar treatment