Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Add Merlin concepts information #731

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

WIP: Add Merlin concepts information #731

wants to merge 1 commit into from

Conversation

mikemckiernan
Copy link
Member

Define the important terms of the DAG.

One important source of information: https://nvidia-merlin.github.io/core/main/api/index.html

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-731

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit ee0b9f24fa965e1409bb92f0b6136173f2c619f8, no merge conflicts.
Running as SYSTEM
Setting status of ee0b9f24fa965e1409bb92f0b6136173f2c619f8 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/615/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse ee0b9f24fa965e1409bb92f0b6136173f2c619f8^{commit} # timeout=10
Checking out Revision ee0b9f24fa965e1409bb92f0b6136173f2c619f8 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ee0b9f24fa965e1409bb92f0b6136173f2c619f8 # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk bae3ccb8c0509ce844a280228943b5e1c73c49bb # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins15860298526554598114.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.8,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.8,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4160785107'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-twpyciv4
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-twpyciv4
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 117.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92208 sha256=647da03e5f30b38d71eccd4d12cd266fe40c90cc2c24036b6f401f324c207368
  Stored in directory: /tmp/pip-ephem-wheel-cache-1wwbgezt/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-prssiubr
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-prssiubr
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 04597b9277b49c8470aa389f77b7e86eeab5d1a9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+42.g04597b92) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.2.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+42.g04597b92) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+42.g04597b92-py3-none-any.whl size=364940 sha256=2d46fc0c02be287085875a2ce53d1452edc2760999cd842151269605be78acf9
  Stored in directory: /tmp/pip-ephem-wheel-cache-wta_nf3x/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+42.g04597b92
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-pjyr5mgy
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-pjyr5mgy
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Building wheels for collected packages: nvtabular
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257596 sha256=58f69c90de14eb2012526384d2de3df140d49f99039249c4636baa8ce82783a2
  Stored in directory: /tmp/pip-ephem-wheel-cache-3amnpntz/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-m_sr71ev
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-m_sr71ev
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit eb606d54fa2ddcbb7e4d0e6501ab2eb418c7fba9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+11.geb606d5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+11.geb606d5-py3-none-any.whl size=118619 sha256=7f41e168db5243e66cca001d687a741ef63f6b56ba38959d7dc81ad3bae69511
  Stored in directory: /tmp/pip-ephem-wheel-cache-b3eks9qs/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+11.geb606d5
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    215    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    146    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               756    340    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     53    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                                57     13    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      4    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     27    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     53    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79     32    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     22    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           54      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          119     67    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 14817   6921    53%


======================== 3 passed in 338.80s (0:05:38) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins8565603947428675334.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit 5470be37ee0ae1c463b6101a3efc81779087467b, no merge conflicts.
Running as SYSTEM
Setting status of 5470be37ee0ae1c463b6101a3efc81779087467b to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/634/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse 5470be37ee0ae1c463b6101a3efc81779087467b^{commit} # timeout=10
Checking out Revision 5470be37ee0ae1c463b6101a3efc81779087467b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5470be37ee0ae1c463b6101a3efc81779087467b # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 569a99c8ceb1b85db960ee601127e51a05eabf94 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins11454089186025869004.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.12,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.12,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@21117cfc4c113b30036afcb97b6daa5f377996db,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3562382237'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-o0g0532k
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-o0g0532k
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 10cd31c23ea417817b827f8695493bdd26ea0948
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+17.g10cd31c) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+17.g10cd31c) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+17.g10cd31c) (1.1.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.1.2)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 95.6 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+17.g10cd31c-py3-none-any.whl size=94038 sha256=c51bfcf4619e49ff99990345bbd12bca729eb42f7eb664837251772dbc13d971
  Stored in directory: /tmp/pip-ephem-wheel-cache-sbmcqvj4/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+17.g10cd31c
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-c0a03xlu
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-c0a03xlu
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit f0805ad747c1ac783e60068ab22fb4980020c090
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+47.gf0805ad7) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+47.gf0805ad7-py3-none-any.whl size=365946 sha256=e076b1bcb2dc88daabed5b96c536b7c82d3f11aa8e2ecf8bea0eb29df469cd9a
  Stored in directory: /tmp/pip-ephem-wheel-cache-1p5ab72_/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+47.gf0805ad7
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-w_s9mng4
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-w_s9mng4
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 21117cfc4c113b30036afcb97b6daa5f377996db
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+5.g21117cfc) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+5.g21117cfc) (1.8.1)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+5.g21117cfc) (0.0.2)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.19.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+5.g21117cfc) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.0.1)
Building wheels for collected packages: nvtabular
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+5.g21117cfc-cp38-cp38-linux_x86_64.whl size=257596 sha256=f87e73aaa6385412e6bab7a12d577927e3ff93c54b5d98d6f2cbadacfc9d1c84
  Stored in directory: /tmp/pip-ephem-wheel-cache-yjmm1o52/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+5.g21117cfc
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-9rcv2fd_
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-9rcv2fd_
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit b298635ce3991007a4961896e21779f6a45348e0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (4.64.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+14.gb298635) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+14.gb298635-py3-none-any.whl size=118651 sha256=b3dff38ee48fff8d6f315366a51e1a27514ef179c24f17c7ab3c052ae19f959c
  Stored in directory: /tmp/pip-ephem-wheel-cache-riuw7pu1/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+14.gb298635
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    215    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    146    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    161    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               756    340    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     53    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      4    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     27    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     46    61%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79     32    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     22    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           54      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 14898   6975    53%


======================== 3 passed in 339.74s (0:05:39) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.20979.265376': database disk image is malformed

data._warn(str(exc))

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.20984.618768' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

data._warn(str(exc))

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins8155960852566438438.sh

@mikemckiernan mikemckiernan changed the title WIP: Add information about the Merlin DAG WIP: Add Merlin concepts information Nov 18, 2022
@mikemckiernan mikemckiernan added the documentation Improvements or additions to documentation label Nov 18, 2022
@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit 8e953fea1fff39b6ce3f3f781d2fc8f95e6a8b8e, no merge conflicts.
Running as SYSTEM
Setting status of 8e953fea1fff39b6ce3f3f781d2fc8f95e6a8b8e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/635/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse 8e953fea1fff39b6ce3f3f781d2fc8f95e6a8b8e^{commit} # timeout=10
Checking out Revision 8e953fea1fff39b6ce3f3f781d2fc8f95e6a8b8e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e953fea1fff39b6ce3f3f781d2fc8f95e6a8b8e # timeout=10
Commit message: "Add stub topics"
 > git rev-list --no-walk 5470be37ee0ae1c463b6101a3efc81779087467b # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins4623343208128903093.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.12,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.12,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@21117cfc4c113b30036afcb97b6daa5f377996db,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3791959737'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-u3yo5vu4
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-u3yo5vu4
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 10cd31c23ea417817b827f8695493bdd26ea0948
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+17.g10cd31c) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+17.g10cd31c) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+17.g10cd31c) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.0.4)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 89.5 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+17.g10cd31c) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+17.g10cd31c-py3-none-any.whl size=94038 sha256=6fafea8dc1021d634b6df5cba7409c003e6c9878e68371b67c48ade64d2be442
  Stored in directory: /tmp/pip-ephem-wheel-cache-_esr_bpv/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+17.g10cd31c
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-ci3ts9td
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-ci3ts9td
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit f0805ad747c1ac783e60068ab22fb4980020c090
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+47.gf0805ad7) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+47.gf0805ad7) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+47.gf0805ad7-py3-none-any.whl size=365946 sha256=938ee54dc610472d21e984c58b0876727d2b1b82beb36ee7aefd337ad4675e39
  Stored in directory: /tmp/pip-ephem-wheel-cache-pxreoucj/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+47.gf0805ad7
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-aphqy6_i
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-aphqy6_i
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 21117cfc4c113b30036afcb97b6daa5f377996db
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+5.g21117cfc) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+5.g21117cfc) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+5.g21117cfc) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.64.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+5.g21117cfc) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+5.g21117cfc) (4.0.0)
Building wheels for collected packages: nvtabular
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+5.g21117cfc-cp38-cp38-linux_x86_64.whl size=257596 sha256=0105c166e9de1d272e613074b220ec287b15960caf0c332862b1db2f9a732c27
  Stored in directory: /tmp/pip-ephem-wheel-cache-q3mvpjw4/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+5.g21117cfc
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-u5twdiup
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-u5twdiup
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit b298635ce3991007a4961896e21779f6a45348e0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+14.gb298635) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+14.gb298635) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+14.gb298635) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+14.gb298635) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+14.gb298635) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+14.gb298635) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+14.gb298635) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+14.gb298635) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+14.gb298635-py3-none-any.whl size=118651 sha256=6b78e431f90f6ae382f670f45ff6fcdcdd2ea26d24a0384fdcf7aa7c37c77bf5
  Stored in directory: /tmp/pip-ephem-wheel-cache-n0sald10/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+14.gb298635
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    161    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               756    340    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     52    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     19    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           54      4    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 14943   6877    54%


======================== 3 passed in 392.96s (0:06:32) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins10262818074499467176.sh

@mikemckiernan mikemckiernan marked this pull request as draft November 24, 2022 23:01
@mikemckiernan mikemckiernan force-pushed the docs-dag-overview branch 2 times, most recently from dbb0b19 to 1f4e38a Compare November 30, 2022 02:36
@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit dbb0b1985a65f75d28d1a5de3e5eef177860692a, no merge conflicts.
Running as SYSTEM
Setting status of dbb0b1985a65f75d28d1a5de3e5eef177860692a to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/662/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse dbb0b1985a65f75d28d1a5de3e5eef177860692a^{commit} # timeout=10
Checking out Revision dbb0b1985a65f75d28d1a5de3e5eef177860692a (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f dbb0b1985a65f75d28d1a5de3e5eef177860692a # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 31de1933ca0ca5b3854dc76bd7c9cb3d8900da0c # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins1856986786029393054.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='809068582'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-ekgo15wz
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-ekgo15wz
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 7.8 MB/s eta 0:00:00
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 165.7 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 40.4 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=15407789f48acf97be9c0d9a6d090d094b6016699d8d480d04f6e4ea17f1dc33
  Stored in directory: /tmp/pip-ephem-wheel-cache-zy4jbefv/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-j1tgz89_
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-j1tgz89_
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=b6e22968da76fd2f89bf2ecf5e29d184aca2299f8e08b6e3cc44e448c6ea1929
  Stored in directory: /tmp/pip-ephem-wheel-cache-hgz7fbgr/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-_f_p60i2
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-_f_p60i2
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=8eae0431bc166644286cf69f5251ef6a63152957c394a6ffe1389a8ae3fabb2f
  Stored in directory: /tmp/pip-ephem-wheel-cache-kvqq01ey/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=68e62df103caff8257ea4a8376600f237cbca4110a334057ffd18932a0be549a
  Stored in directory: /tmp/pip-ephem-wheel-cache-kvqq01ey/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-u9p6czv8
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-u9p6czv8
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=51b205fa9c277c88451811100d6458aa5d7532379756f23ec7d309122ff921cf
  Stored in directory: /tmp/pip-ephem-wheel-cache-rnoxn_9a/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7f1bf12c6e20>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '3f06bc87-4c78-4de...e, 'engine': '3f06bc87-4c78-4de3-8784-b2327922462f', 'started': '2022-11-30T02:38:10.765800Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-30 02:38:13,756 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:13,768 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:13,782 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:13,836 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:14,234 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:14,235 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:17,032 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-0rwtdnck', purging

2022-11-30 02:38:17,033 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:17,295 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:17,296 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,220 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-j137sm4s', purging

2022-11-30 02:38:20,221 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:20,235 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:20,243 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:20,278 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:20,721 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,722 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,816 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,817 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,822 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:20,823 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:23,447 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-u0mvggab', purging

2022-11-30 02:38:23,447 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-xgq65226', purging

2022-11-30 02:38:23,447 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-bdxeh1xu', purging

2022-11-30 02:38:23,448 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:38:23,723 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:38:23,724 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    158    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     17    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    111    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    161    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     84    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    162    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    357    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    110    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     24    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     40    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     19    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     17    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     96    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%


TOTAL                                                                                                 15014   7167    52%


=================== 1 failed, 2 passed in 234.92s (0:03:54) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins12077160033250436366.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit 1f4e38a2c87df15afca8f333e2e7d604ceb2196a, no merge conflicts.
Running as SYSTEM
Setting status of 1f4e38a2c87df15afca8f333e2e7d604ceb2196a to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/663/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse 1f4e38a2c87df15afca8f333e2e7d604ceb2196a^{commit} # timeout=10
Checking out Revision 1f4e38a2c87df15afca8f333e2e7d604ceb2196a (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1f4e38a2c87df15afca8f333e2e7d604ceb2196a # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk dbb0b1985a65f75d28d1a5de3e5eef177860692a # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins3732260165180736111.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2566426484'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-m0s9fb27
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-m0s9fb27
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.1 MB/s eta 0:00:00
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 15.4 MB/s eta 0:00:00
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 114.2 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=8aae7d76669accde8152b86baeea59a6eaa2005687042118b001a1508d113bc2
  Stored in directory: /tmp/pip-ephem-wheel-cache-3h4j7rct/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-3l8gxic7
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-3l8gxic7
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.3.0+12.g78ecddd)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=dc4f5330e78a4a2ece578ebd725466e4981ef9f3185a3582e8c380e45abe25a5
  Stored in directory: /tmp/pip-ephem-wheel-cache-escqmpee/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-vdzej9cf
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-vdzej9cf
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 799.9 kB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=cc8ebd893d53a22e6c57aebf49c4308d5f570be9c8450a89dc3a908cc179de3c
  Stored in directory: /tmp/pip-ephem-wheel-cache-3x35jdvq/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=04475c4a5590b06aaf2212e875efc35f16fcd23cb733fcbee81d29ad1c579567
  Stored in directory: /tmp/pip-ephem-wheel-cache-3x35jdvq/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-smmc7gnl
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-smmc7gnl
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=8ee640cfc80b16d35c88faac7217e19dd768174aec08871c06fe2fddbc0814c5
  Stored in directory: /tmp/pip-ephem-wheel-cache-r4aeuhn7/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7f80dc2fad90>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7f80dc2fad90>, {'id': 'f2997aee', 'cell_type': 'code', 'metadata'...[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}, 55)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7f80db772f40>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)\nRuntimeError: Tritonserver failed to start (ret=1)\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7f80dc2fad90>

cell = {'id': 'f2997aee', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-30T02:41:46.783905Z',...t\x1b[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}

cell_index = 55, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7f80dc2fad90>

cell = {'id': 'f2997aee', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-30T02:41:46.783905Z',...t\x1b[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '6bf878e3-f801-4e...e, 'engine': '6bf878e3-f801-4ea0-ab74-2fe3ae929f01', 'started': '2022-11-30T02:41:46.784290Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E           �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E           �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E           �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E           �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E           �[1;32m    141�[0m     )

E           �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E           �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E           �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E           �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E           �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:81�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E           �[1;32m     79�[0m �[38;5;28;01mif�[39;00m process�[38;5;241m.�[39mpoll() �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[1;32m     80�[0m     retcode �[38;5;241m=�[39m process�[38;5;241m.�[39mreturncode

E           �[0;32m---> 81�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mTritonserver failed to start (ret=�[39m�[38;5;132;01m{�[39;00mretcode�[38;5;132;01m}�[39;00m�[38;5;124m)�[39m�[38;5;124m"�[39m)

E           �[1;32m     83�[0m �[38;5;28;01mtry�[39;00m:

E           �[1;32m     84�[0m     ready �[38;5;241m=�[39m client�[38;5;241m.�[39mis_server_ready()

E

E           �[0;31mRuntimeError�[0m: Tritonserver failed to start (ret=1)

E           RuntimeError: Tritonserver failed to start (ret=1)


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"





      tb2.inject(





            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )



tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7f80dc2fad90>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E               �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E               �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E               �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E               �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E               �[1;32m    141�[0m     )

E               �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E               �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E               �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E               �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E               �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E               �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:81�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E               �[1;32m     79�[0m �[38;5;28;01mif�[39;00m process�[38;5;241m.�[39mpoll() �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E               �[1;32m     80�[0m     retcode �[38;5;241m=�[39m process�[38;5;241m.�[39mreturncode

E               �[0;32m---> 81�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mTritonserver failed to start (ret=�[39m�[38;5;132;01m{�[39;00mretcode�[38;5;132;01m}�[39;00m�[38;5;124m)�[39m�[38;5;124m"�[39m)

E               �[1;32m     83�[0m �[38;5;28;01mtry�[39;00m:

E               �[1;32m     84�[0m     ready �[38;5;241m=�[39m client�[38;5;241m.�[39mis_server_ready()

E

E               �[0;31mRuntimeError�[0m: Tritonserver failed to start (ret=1)

E               RuntimeError: Tritonserver failed to start (ret=1)


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-11-30 02:40:32.150944: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-30 02:40:36.325660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:40:36.326472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-30 02:40:36.327129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-30 02:40:36.327702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 457 points to 32 centroids: please provide at least 1248 training points

2022-11-30 02:41:40.144901: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-30 02:41:44.190503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:41:44.191241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-30 02:41:44.192723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-30 02:41:44.193779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1130 02:41:47.102713 18626 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe686000000' with size 268435456

I1130 02:41:47.103434 18626 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1130 02:41:47.111362 18626 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1130 02:41:47.111413 18626 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1130 02:41:47.111472 18626 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1130 02:41:47.111506 18626 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1130 02:41:47.111559 18626 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1130 02:41:47.111599 18626 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1130 02:41:47.111665 18626 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

I1130 02:42:55.111481 18626 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1130 02:42:55.111530 18626 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1130 02:42:55.111537 18626 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1130 02:42:55.111543 18626 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1130 02:42:55.111584 18626 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)

I1130 02:43:01.874680 18626 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

I1130 02:43:01.874837 18626 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1

2022-11-30 02:43:01.876299: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:01.888299: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-30 02:43:01.888332: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:01.888448: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-30 02:43:01.931091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5144 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:43:01.972755: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-11-30 02:43:01.974803: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-30 02:43:02.054432: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:02.079309: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 203040 microseconds.

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1130 02:43:18.340642 18626 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)

2022-11-30 02:43:18.341973: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:18.362408: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-30 02:43:18.362451: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:18.364530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5144 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:43:18.394123: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-30 02:43:18.532981: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:18.585224: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 243263 microseconds.

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1130 02:43:34.834529 18626 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-11-30 02:43:34.835784: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:34.844668: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-30 02:43:34.844707: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:34.847300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5144 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:43:34.857887: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-30 02:43:34.918149: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:34.944155: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 108385 microseconds.

I1130 02:43:34.944283 18626 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)

I1130 02:43:34.944497 18626 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

I1130 02:43:41.360605 18626 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)

I1130 02:43:41.361722 18626 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1

I1130 02:43:47.731532 18626 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)

I1130 02:43:47.731841 18626 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1

2022-11-30 02:43:47.732504: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:47.751612: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-30 02:43:47.751660: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:47.753779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5144 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 02:43:47.778079: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-30 02:43:47.920375: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-30 02:43:47.970960: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 238466 microseconds.

I1130 02:43:47.971152 18626 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)

I1130 02:43:47.971360 18626 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1

I1130 02:43:54.181958 18626 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)

I1130 02:43:54.182290 18626 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1

I1130 02:44:00.397843 18626 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1

E1130 02:44:00.398057 18626 model_repository_manager.cc:481] Invalid argument: in ensemble ensemble_model, ensemble tensor user_id_raw is mapping to non-existing input user_id_raw in model 0_queryfeast

I1130 02:44:00.398202 18626 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1130 02:44:00.398330 18626 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1130 02:44:00.398451 18626 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_queryfeast              | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| 2_queryfaiss              | 1       | READY  |

| 3_queryfeast              | 1       | READY  |

| 4_unrollfeatures          | 1       | READY  |

| 5_predicttensorflowtriton | 1       | READY  |

| 6_softmaxsampling         | 1       | READY  |

+---------------------------+---------+--------+


I1130 02:44:00.464414 18626 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1130 02:44:00.465259 18626 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/examples/poc_ensemble                                                                                                                                                                   |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1130 02:44:00.465309 18626 server.cc:262] Waiting for in-flight requests to complete.

I1130 02:44:00.465328 18626 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1130 02:44:00.465565 18626 server.cc:293] All models are stopped, unloading models

I1130 02:44:00.465579 18626 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests

I1130 02:44:00.465571 18626 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1130 02:44:00.465619 18626 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1130 02:44:00.465922 18626 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1130 02:44:00.465985 18626 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1130 02:44:00.474379 18626 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1130 02:44:00.494045 18626 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1

I1130 02:44:01.465731 18626 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests

W1130 02:44:01.487821 18626 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1130 02:44:02.465931 18626 server.cc:300] Timeout 28: Found 5 live models and 0 in-flight non-inference requests

W1130 02:44:02.488044 18626 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1130 02:44:02.518670 18626 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1

I1130 02:44:02.579279 18626 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1

I1130 02:44:02.611296 18626 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1

I1130 02:44:02.635536 18626 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1

I1130 02:44:02.662363 18626 model_lifecycle.cc:578] successfully unloaded '0_queryfeast' version 1

I1130 02:44:03.466056 18626 server.cc:300] Timeout 27: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1130 02:44:03.514401 18626 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7f81b9545fd0>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '713d04f2-5a37-47e...e, 'engine': '713d04f2-5a37-47ee-b1f3-fb60887748ce', 'started': '2022-11-30T02:44:14.081287Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-30 02:44:17,049 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:17,054 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:17,071 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:17,169 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:17,601 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:17,602 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:20,309 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-b9d1089d', purging

2022-11-30 02:44:20,310 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:20,571 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:20,572 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:23,409 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-pzzh0cqd', purging

2022-11-30 02:44:23,410 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:23,423 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:23,439 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:23,446 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:23,980 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:23,981 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:23,989 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:23,990 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:24,024 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:24,025 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:24,032 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:24,033 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:26,621 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-0vg_a74a', purging

2022-11-30 02:44:26,621 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-jexanjio', purging

2022-11-30 02:44:26,622 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-gf1yvtxu', purging

2022-11-30 02:44:26,622 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-q_d9mtv2', purging

2022-11-30 02:44:26,623 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 02:44:26,892 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 02:44:26,893 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    158    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     17    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    111    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    161    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     84    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    162    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    357    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    110    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     24    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     40    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     74    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      7    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     36    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     11    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     10    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     34    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143    120    16%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     25    65%


TOTAL                                                                                                 15014   7291    51%


=================== 2 failed, 1 passed in 254.56s (0:04:14) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins14865758638763260489.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit ef0442296c4eeeed3dc57a2836eb5e03a5cbf391, no merge conflicts.
Running as SYSTEM
Setting status of ef0442296c4eeeed3dc57a2836eb5e03a5cbf391 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/664/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse ef0442296c4eeeed3dc57a2836eb5e03a5cbf391^{commit} # timeout=10
Checking out Revision ef0442296c4eeeed3dc57a2836eb5e03a5cbf391 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ef0442296c4eeeed3dc57a2836eb5e03a5cbf391 # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 1f4e38a2c87df15afca8f333e2e7d604ceb2196a # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins10285209716803419990.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='849401712'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-7fo_jn7o
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-7fo_jn7o
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 3.0 MB/s eta 0:00:00
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 13.8 MB/s eta 0:00:00
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 87.1 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=f46abf77718ddaca9aadbb4ebc9e0873aeff7b53de4674072f937b9028fecc9c
  Stored in directory: /tmp/pip-ephem-wheel-cache-dlk2i10r/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-lac6ptvf
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-lac6ptvf
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=f924f7a555376403dead6380ae904bfd3ed85a573dace710c956b7298b6f4009
  Stored in directory: /tmp/pip-ephem-wheel-cache-snn2czzt/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-1xft3hru
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-1xft3hru
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=db27f522d4ef41f216f4397f3cb303f75dfe16516a7a51b4f0c5652296b73f78
  Stored in directory: /tmp/pip-ephem-wheel-cache-fqt6c1ny/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=666c9be61e7269699853b65418a0dd658b53042063cc9df6b794f0f88be50275
  Stored in directory: /tmp/pip-ephem-wheel-cache-fqt6c1ny/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ect19wez
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ect19wez
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=c832fce1a6ec20e28f8a8c4a30f3ee08152ca6ffdf2d29647ad163bc5d5e96ae
  Stored in directory: /tmp/pip-ephem-wheel-cache-d4_rx1wr/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    108    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    342    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     33    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     57    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     19    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      4    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 15293   7063    54%


======================== 3 passed in 328.32s (0:05:28) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.8530.938535' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

data._warn(str(exc))

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins10433080166375599886.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit 8ff535772a2b573af8968ed7d1ee0940f5211e81, no merge conflicts.
Running as SYSTEM
Setting status of 8ff535772a2b573af8968ed7d1ee0940f5211e81 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/665/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse 8ff535772a2b573af8968ed7d1ee0940f5211e81^{commit} # timeout=10
Checking out Revision 8ff535772a2b573af8968ed7d1ee0940f5211e81 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8ff535772a2b573af8968ed7d1ee0940f5211e81 # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk ef0442296c4eeeed3dc57a2836eb5e03a5cbf391 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins815829709532640159.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3018476971'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-ainmlevp
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-ainmlevp
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit c73e81898c4c3ba10df019b9f36fa409a39d2626
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (1.1.1)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 2.2 MB/s eta 0:00:00
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 6.1 MB/s eta 0:00:00
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.gc73e818) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.gc73e818) (2.22.0)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+21.gc73e818) (1.8.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.4)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 37.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.gc73e818) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.gc73e818-py3-none-any.whl size=98087 sha256=ed49888051d84e464c2850823fbb7d405c979d769fab3108ef8b987db2e66c15
  Stored in directory: /tmp/pip-ephem-wheel-cache-0xrtzyqs/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+21.gc73e818 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-jp08mtuk
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-jp08mtuk
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 5e4d8a106498847b5755f94a8e467c46cbf12c60
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+60.g5e4d8a10) (0.3.0+12.g78ecddd)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+60.g5e4d8a10-py3-none-any.whl size=366424 sha256=d1a7e08c4db116d0b1e9adfc9c918caec92b31d6fc22254260498f768b02cfc4
  Stored in directory: /tmp/pip-ephem-wheel-cache-rn1b37t1/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+60.g5e4d8a10
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-vf525ico
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-vf525ico
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ee3b7440eb22fad71ad50a6df978e1fa526f4817
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+11.gee3b7440) (0.3.0+12.g78ecddd)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+11.gee3b7440) (1.8.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+11.gee3b7440) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+11.gee3b7440) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+11.gee3b7440-cp38-cp38-linux_x86_64.whl size=257603 sha256=5c8bb08beed50b105c2003b25e9c68825c6325cb112e7fa3614f73300268e6c2
  Stored in directory: /tmp/pip-ephem-wheel-cache-bwsoh06r/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=8aed29dd5f94e1550b11f592f494987a716e941095977d65239a6cd5a1c94fb6
  Stored in directory: /tmp/pip-ephem-wheel-cache-bwsoh06r/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+11.gee3b7440
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-b0w5sqs9
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-b0w5sqs9
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=0ab18beb6514711b675997eb43f40f3bbc66192d80ee9de443600bee945a0ae2
  Stored in directory: /tmp/pip-ephem-wheel-cache-q_g_69fg/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7f6ed593b2b0>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7f6ed593b2b0>, {'id': '0d415879', 'cell_type': 'code', 'metadata'...pass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}, 55)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7f6ed587cf40>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...rver is already running on localhost:8001\nRuntimeError: Another tritonserver is already running on localhost:8001\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7f6ed593b2b0>

cell = {'id': '0d415879', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-30T21:00:51.701553Z',...8;01mpass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}

cell_index = 55, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7f6ed593b2b0>

cell = {'id': '0d415879', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-30T21:00:51.701553Z',...8;01mpass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '7ab94c61-b27f-43...e, 'engine': '7ab94c61-b27f-43de-b651-0fd219723c34', 'started': '2022-11-30T21:00:51.701910Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E           �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E           �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E           �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E           �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E           �[1;32m    141�[0m     )

E           �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E           �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E           �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E           �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E           �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:58�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E           �[1;32m     56�[0m     �[38;5;28;01mwith�[39;00m grpcclient�[38;5;241m.�[39mInferenceServerClient(grpc_url) �[38;5;28;01mas�[39;00m client:

E           �[1;32m     57�[0m         �[38;5;28;01mif�[39;00m client�[38;5;241m.�[39mis_server_ready():

E           �[0;32m---> 58�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mAnother tritonserver is already running on �[39m�[38;5;132;01m{�[39;00mgrpc_url�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m)

E           �[1;32m     59�[0m �[38;5;28;01mexcept�[39;00m tritonclient�[38;5;241m.�[39mutils�[38;5;241m.�[39mInferenceServerException:

E           �[1;32m     60�[0m     �[38;5;28;01mpass�[39;00m

E

E           �[0;31mRuntimeError�[0m: Another tritonserver is already running on localhost:8001

E           RuntimeError: Another tritonserver is already running on localhost:8001


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"





      tb2.inject(





            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )



tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7f6ed593b2b0>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E               �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E               �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E               �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E               �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E               �[1;32m    141�[0m     )

E               �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E               �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E               �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E               �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E               �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E               �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:58�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E               �[1;32m     56�[0m     �[38;5;28;01mwith�[39;00m grpcclient�[38;5;241m.�[39mInferenceServerClient(grpc_url) �[38;5;28;01mas�[39;00m client:

E               �[1;32m     57�[0m         �[38;5;28;01mif�[39;00m client�[38;5;241m.�[39mis_server_ready():

E               �[0;32m---> 58�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mAnother tritonserver is already running on �[39m�[38;5;132;01m{�[39;00mgrpc_url�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m)

E               �[1;32m     59�[0m �[38;5;28;01mexcept�[39;00m tritonclient�[38;5;241m.�[39mutils�[38;5;241m.�[39mInferenceServerException:

E               �[1;32m     60�[0m     �[38;5;28;01mpass�[39;00m

E

E               �[0;31mRuntimeError�[0m: Another tritonserver is already running on localhost:8001

E               RuntimeError: Another tritonserver is already running on localhost:8001


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-11-30 20:59:16.968989: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-30 20:59:21.100819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 20:59:21.101571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-30 20:59:21.102234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-30 20:59:21.102864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 441 points to 32 centroids: please provide at least 1248 training points

2022-11-30 21:00:44.932228: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-30 21:00:49.023614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-30 21:00:49.024917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13875 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-30 21:00:49.026563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 13875 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-30 21:00:49.027158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 13875 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7f6fb3e0daf0>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'e3a886b4-bd82-486...e, 'engine': 'e3a886b4-bd82-4863-877e-e87f1a1ebe91', 'started': '2022-11-30T21:01:01.453037Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-30 21:01:04,489 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:04,509 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:04,517 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:04,535 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:05,051 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:05,052 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:07,816 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-ja9scb6j', purging

2022-11-30 21:01:07,817 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:08,095 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:08,096 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:10,894 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-4r40d9q7', purging

2022-11-30 21:01:10,894 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:10,901 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:10,916 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:10,985 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:11,457 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,458 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,479 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,479 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,548 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,549 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,597 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:11,599 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:14,197 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-0a8ymqkl', purging

2022-11-30 21:01:14,198 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-6ydxgto5', purging

2022-11-30 21:01:14,198 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-8w8dp96a', purging

2022-11-30 21:01:14,198 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-7fw1pvsf', purging

2022-11-30 21:01:14,199 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-30 21:01:14,469 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-30 21:01:14,470 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 42 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    163    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     17    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    111    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    161    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     84    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    162    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    357    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    110    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     24    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     40    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     74    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     23    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     54    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     17    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     15    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     34    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143    120    16%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     45    38%


TOTAL                                                                                                 15014   7373    51%


=================== 2 failed, 1 passed in 136.99s (0:02:16) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins1774250875501254762.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit d1c3e5e44843938bdb221bac17cfedced3ac040b, no merge conflicts.
Running as SYSTEM
Setting status of d1c3e5e44843938bdb221bac17cfedced3ac040b to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/666/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_merlin/workspace
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse d1c3e5e44843938bdb221bac17cfedced3ac040b^{commit} # timeout=10
Checking out Revision d1c3e5e44843938bdb221bac17cfedced3ac040b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d1c3e5e44843938bdb221bac17cfedced3ac040b # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 8ff535772a2b573af8968ed7d1ee0940f5211e81 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins10627132525495257850.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1996603426'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-xx0pqonw
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-xx0pqonw
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit a849a806fcefcadaabe4bb24ff89373263ffce09
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 8.5 MB/s eta 0:00:00
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.ga849a80) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+22.ga849a80) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.ga849a80) (1.1.1)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 109.8 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.ga849a80) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.ga849a80) (1.20.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (7.0.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 114.9 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.ga849a80) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+22.ga849a80-py3-none-any.whl size=98126 sha256=466ebf8205592c9f29ad059409fa3ce569d7b1c8770d539c16b2e571ad14ead3
  Stored in directory: /tmp/pip-ephem-wheel-cache-l7r57m4t/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+22.ga849a80 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-tpkck_8u
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-tpkck_8u
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 5e4d8a106498847b5755f94a8e467c46cbf12c60
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+60.g5e4d8a10) (0.3.0+12.g78ecddd)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (7.0.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+60.g5e4d8a10-py3-none-any.whl size=366424 sha256=2da6534f150e9deffae299aa993a1dfaa4cd09ee4f7f8c9a166ed117cb3b119c
  Stored in directory: /tmp/pip-ephem-wheel-cache-3_4f21qu/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+60.g5e4d8a10
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-pjdlswhm
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-pjdlswhm
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 3bcd555a7fdc1cdaca47379676c2f111750352a8
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+12.g3bcd555a) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+12.g3bcd555a) (1.8.1)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.7 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (7.0.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+12.g3bcd555a) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+12.g3bcd555a-cp38-cp38-linux_x86_64.whl size=257601 sha256=3b57b2951d385d6b4bfbd7f1b92a190ffa3b24d6f2e95ef7342131cfeb2cb040
  Stored in directory: /tmp/pip-ephem-wheel-cache-bbz2qb8p/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=142fca2ef2c1e710148193bf722c9763ae8b322fc18647727061c1bd9db5d7ff
  Stored in directory: /tmp/pip-ephem-wheel-cache-bbz2qb8p/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+12.g3bcd555a
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-pqgc12ez
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-pqgc12ez
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=d70af391cbb57a51b4d3ae2442ad2f52532c5fe58e3f6a592d753bcd25349550
  Stored in directory: /tmp/pip-ephem-wheel-cache-igwx9hwb/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          624    356    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    108    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              175    121    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               758    342    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     33    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     57    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     49    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     19    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             65     19    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      4    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 145     47    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143     74    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 15314   7075    54%


======================== 3 passed in 293.94s (0:04:53) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins10737108515133556771.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit 787e88c486208b30f600428cb64a4fcfb85d73e0, no merge conflicts.
Running as SYSTEM
Setting status of 787e88c486208b30f600428cb64a4fcfb85d73e0 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/669/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_merlin/workspace
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse 787e88c486208b30f600428cb64a4fcfb85d73e0^{commit} # timeout=10
Checking out Revision 787e88c486208b30f600428cb64a4fcfb85d73e0 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 787e88c486208b30f600428cb64a4fcfb85d73e0 # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 84c9f5de5fd846a95d252d6a3ecf63ddfe6f0fb4 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins5056576637564943171.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.21,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.21,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1770026436'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-9gyhvq7m
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-9gyhvq7m
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 4f3d92dc259971eee2fee319713b0d7a08a57c54
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+24.g4f3d92d) (1.1.1)
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 7.2 MB/s eta 0:00:00
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+24.g4f3d92d) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+24.g4f3d92d) (0.3.0+12.g78ecddd)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 144.4 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+24.g4f3d92d) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+24.g4f3d92d) (1.20.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 95.6 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+24.g4f3d92d-py3-none-any.whl size=100465 sha256=2fe4ecfb4216a1d3ee59c4a828f3af4cb89f5da68bfaa56e1657e0e82ad1dd1b
  Stored in directory: /tmp/pip-ephem-wheel-cache-4b00r1qe/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+24.g4f3d92d treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-hff9hh9m
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-hff9hh9m
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit e08a72c9c59416a9000e62d25548eb08367fc3fa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+61.ge08a72c9) (0.3.0+12.g78ecddd)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+61.ge08a72c9-py3-none-any.whl size=367208 sha256=22e80fc1f106712773424e448c7e86da329795182a4bbaa900c31f5b6a73e84b
  Stored in directory: /tmp/pip-ephem-wheel-cache-borxx1cx/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+61.ge08a72c9
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-sqe_3vec
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-sqe_3vec
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 51af616069689b3ba57e8842a6f4a20377795df7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+13.g51af6160) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+13.g51af6160) (0.3.0+12.g78ecddd)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.10.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+13.g51af6160) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+13.g51af6160-cp38-cp38-linux_x86_64.whl size=257601 sha256=ce12d19d7ef69bfb36d92d1b6770cd17460d5df8aa587e284a69776f4b5c2508
  Stored in directory: /tmp/pip-ephem-wheel-cache-aaz62vjx/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=4011d2235aed25e6f41672f6dd6c99852cf0f8a36dc2ba27ef337a1c15986724
  Stored in directory: /tmp/pip-ephem-wheel-cache-aaz62vjx/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+13.g51af6160
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-twb9ctfn
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-twb9ctfn
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=46b2ed2d9ae6f6f338b4bd6a477d89445c20265d9cb3bf54c4834a21ae65470c
  Stored in directory: /tmp/pip-ephem-wheel-cache-aefl1rhi/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7f07a560cd60>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7f07a560cd60>, {'id': '57bb182e', 'cell_type': 'code', 'metadata'...on\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}, 55)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7f07a4a84ec0>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...rException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7f07a560cd60>

cell = {'id': '57bb182e', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:20:55.134324Z',...ception\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}

cell_index = 55, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7f07a560cd60>

cell = {'id': '57bb182e', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:20:55.134324Z',...ception\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'InferenceServerException', 'engine_info': {'engine_id': -1, 'engine_uuid': '61ec...e, 'engine': '61ec042d-362d-48d0-bca7-2ef94f6e6c47', 'started': '2022-12-02T14:20:55.134649Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mInferenceServerException�[0m                  Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:140�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    138�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    139�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[0;32m--> 140�[0m     response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m

E           �[1;32m    141�[0m �[43m        �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m

E           �[1;32m    142�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m    144�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:193�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m

E           �[1;32m    191�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]

E           �[1;32m    192�[0m �[38;5;28;01mwith�[39;00m client:

E           �[0;32m--> 193�[0m     response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m

E           �[1;32m    195�[0m results �[38;5;241m=�[39m {}

E           �[1;32m    196�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m

E           �[1;32m   1320�[0m     �[38;5;28;01mreturn�[39;00m result

E           �[1;32m   1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:

E           �[0;32m-> 1322�[0m     �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m

E           �[1;32m     61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):

E           �[0;32m---> 62�[0m     �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           �[0;31mInferenceServerException�[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'

E           InferenceServerException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"





      tb2.inject(





            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )



tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7f07a560cd60>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mInferenceServerException�[0m                  Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:140�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    138�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[1;32m    139�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[0;32m--> 140�[0m     response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m

E               �[1;32m    141�[0m �[43m        �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m

E               �[1;32m    142�[0m �[43m    �[49m�[43m)�[49m

E               �[1;32m    144�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:193�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m

E               �[1;32m    191�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]

E               �[1;32m    192�[0m �[38;5;28;01mwith�[39;00m client:

E               �[0;32m--> 193�[0m     response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m

E               �[1;32m    195�[0m results �[38;5;241m=�[39m {}

E               �[1;32m    196�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:

E

E               File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m

E               �[1;32m   1320�[0m     �[38;5;28;01mreturn�[39;00m result

E               �[1;32m   1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:

E               �[0;32m-> 1322�[0m     �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m

E

E               File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m

E               �[1;32m     61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):

E               �[0;32m---> 62�[0m     �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               �[0;31mInferenceServerException�[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'

E               InferenceServerException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-02 14:19:36.497401: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:19:40.640435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:19:40.641184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:19:40.641799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:19:40.642438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 464 points to 32 centroids: please provide at least 1248 training points

2022-12-02 14:20:48.086338: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:20:52.219131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:20:52.220001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:20:52.220587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:20:52.221187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1202 14:20:55.450064 5001 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f1124000000' with size 268435456

I1202 14:20:55.450795 5001 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1202 14:20:55.459019 5001 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1202 14:20:55.459077 5001 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1202 14:20:55.459116 5001 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1202 14:20:55.459160 5001 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1202 14:20:55.459221 5001 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1202 14:20:55.459259 5001 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1202 14:20:55.459353 5001 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

I1202 14:20:55.748959 5001 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1202 14:20:55.749020 5001 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1202 14:20:55.749028 5001 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1202 14:20:55.749035 5001 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1202 14:20:55.749077 5001 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

2022-12-02 14:20:55.749702: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:55.772558: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-12-02 14:20:55.772610: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:55.772748: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:20:56.212452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2568 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:20:56.239267: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-12-02 14:20:56.241290: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-12-02 14:20:56.294716: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:56.318733: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 569042 microseconds.

I1202 14:20:56.329267 5001 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-12-02 14:20:56.329781: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:56.333042: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-12-02 14:20:56.333068: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:56.334971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2568 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:20:56.342317: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-12-02 14:20:56.399652: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:20:56.424579: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 94803 microseconds.

I1202 14:20:56.424884 5001 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

1202 14:21:01.674558 5079 pb_stub.cc:1016] Non-graceful termination detected.

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7f088327c220>

msg_id = 'c2711541-b750ade5a7b694f708b78626_3795_14'

cell = {'id': '141ce318', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:22:45.933206Z',...ee("/tmp/input/criteo", ignore_errors=True)\nshutil.rmtree("/tmp/output/criteo", ignore_errors=True)\n', 'outputs': []}

timeout = 180

task_poll_output_msg = <Task pending name='Task-613' coro=<NotebookClient._async_poll_output_msg() running at /var/jenkins_home/.local/lib/py...ite-packages/nbclient/client.py:762> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f08829db670>()]>>

task_poll_kernel_alive = <Task cancelled name='Task-612' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>


async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:





          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))





../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:




obj = <coroutine object ZMQSocketChannel.get_msg at 0x7f087dff9240>


async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:





          result = await obj





../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:




self = <jupyter_client.channels.ZMQSocketChannel object at 0x7f08829b9ca0>

timeout = 180000.0


async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:





      raise Empty





E           _queue.Empty


/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
        tb1.execute()
        assert os.path.isfile("/tmp/output/criteo/train/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/valid/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/workflow/metadata.json")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "03-Training-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["INPUT_DATA_DIR"] = "/tmp/output/criteo/"
            """
        )
        tb2.execute()
        metrics = tb2.ref("eval_metrics")
        assert set(metrics.keys()) == set(
            [
                "auc",
                "binary_accuracy",
                "loss",
                "precision",
                "recall",
                "regularization_loss",
                "loss_batch",
            ]
        )
        assert os.path.isfile("/tmp/output/criteo/dlrm/saved_model.pb")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "04-Triton-Inference-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb3:
        tb3.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/output/criteo/"
            os.environ["INPUT_FOLDER"] = "/tmp/input/criteo/"
            """
        )
        NUM_OF_CELLS = len(tb3.cells)
        tb3.execute_cell(list(range(0, NUM_OF_CELLS - 5)))





      tb3.inject(





            """
            import shutil
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            outputs = ensemble.output_schema.column_names
            response = run_ensemble_on_tritonserver(
                "/tmp/output/criteo/ensemble/",workflow.input_schema, batch.fillna(0),
                outputs, "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["label/binary_classification_task"]]
            shutil.rmtree("/tmp/input/criteo", ignore_errors=True)
            shutil.rmtree("/tmp/output/criteo", ignore_errors=True)
            """
        )



tests/unit/examples/test_scaling_criteo_merlin_models.py:84:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell

cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell

exec_reply = await self.task_poll_for_reply

../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply

await self._async_handle_timeout(timeout, cell)




self = <testbook.client.TestbookNotebookClient object at 0x7f088327c220>

timeout = 180

cell = {'id': '141ce318', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:22:45.933206Z',...ee("/tmp/input/criteo", ignore_errors=True)\nshutil.rmtree("/tmp/output/criteo", ignore_errors=True)\n', 'outputs': []}


async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:





      raise CellTimeoutError.error_from_timeout_and_cell(





            "Cell execution timed out", timeout, cell
        )



E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 180 seconds.

E           The message was: Cell execution timed out.

E           Here is a preview of the cell contents:

E           -------------------

E

E           import shutil

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           outputs = ensemble.output_schema.column_names

E           response = run_ensemble_on_tritonserver(

E               "/tmp/output/criteo/ensemble/",workflow.input_schema, batch.fillna(0),

E               outputs, "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["label/binary_classification_task"]]

E           shutil.rmtree("/tmp/input/criteo", ignore_errors=True)

E           shutil.rmtree("/tmp/output/criteo", ignore_errors=True)

E

E           -------------------


../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (2) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

----------------------------- Captured stderr call -----------------------------

2022-12-02 14:21:13,834 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:21:13,845 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:21:13,852 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:21:13,873 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:21:14,447 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:21:14,447 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:21:17,069 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-udipa3i2', purging

2022-12-02 14:21:17,069 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 27 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

2022-12-02 14:21:36.670737: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:21:40.934280: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-12-02 14:21:40.934395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:21:40.935189: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-12-02 14:21:40.935267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:21:40.935899: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 2

2022-12-02 14:21:40.935948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:21:40.936597: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 3

2022-12-02 14:21:40.936649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

2022-12-02 14:22:17.903326: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:22:22.140821: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-12-02 14:22:22.140920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:22:22.142710: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-12-02 14:22:22.142813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:22:22.143836: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 2

2022-12-02 14:22:22.143937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:22:22.145620: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 3

2022-12-02 14:22:22.145715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1202 14:22:46.302258 6685 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe9c6000000' with size 268435456

I1202 14:22:46.303021 6685 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1202 14:22:46.308580 6685 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1202 14:22:46.308680 6685 model_lifecycle.cc:459] loading: 0_transformworkflowtriton:1

I1202 14:22:46.608318 6685 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1202 14:22:46.608360 6685 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1202 14:22:46.608393 6685 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1202 14:22:46.608399 6685 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1202 14:22:46.608450 6685 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

2022-12-02 14:22:46.609327: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:46.626512: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-12-02 14:22:46.626572: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:46.626704: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:22:47.062485: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-12-02 14:22:47.062598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 11411 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:22:47.130486: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-12-02 14:22:47.137701: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-12-02 14:22:47.327032: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:47.391057: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 781740 microseconds.

I1202 14:22:47.417014 6685 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-12-02 14:22:47.417554: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:47.429843: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-12-02 14:22:47.429892: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:47.431891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 11411 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:22:47.461926: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-12-02 14:22:47.598168: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/output/criteo/ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-12-02 14:22:47.657222: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 239673 microseconds.

I1202 14:22:47.657606 6685 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1202 14:22:56.517129 6685 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflowtriton (GPU device 0)

I1202 14:23:03.820350 6685 model_lifecycle.cc:693] successfully loaded '0_transformworkflowtriton' version 1

I1202 14:23:03.822161 6685 model_lifecycle.cc:459] loading: ensemble_model:1

I1202 14:23:03.823012 6685 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1202 14:23:03.823138 6685 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1202 14:23:03.823254 6685 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1202 14:23:03.823334 6685 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_transformworkflowtriton | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| ensemble_model            | 1       | READY  |

+---------------------------+---------+--------+


I1202 14:23:03.891377 6685 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1202 14:23:03.892285 6685 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/output/criteo/ensemble/                                                                                                                                                                 |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 536870912                                                                                                                                                                                    |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1202 14:23:03.893847 6685 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1202 14:23:03.894056 6685 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1202 14:23:03.934985 6685 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1202 14:23:04.916406 6685 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1202 14:23:05.916594 6685 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1202 14:23:06.935429 6685 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1202 14:25:45.935032 6685 server.cc:262] Waiting for in-flight requests to complete.

I1202 14:25:45.935058 6685 server.cc:278] Timeout 30: Found 1 model versions that have in-flight inferences

I1202 14:25:45.935066 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

I1202 14:25:46.935597 6685 server.cc:278] Timeout 29: Found 1 model versions that have in-flight inferences

I1202 14:25:46.935641 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

I1202 14:25:47.935798 6685 server.cc:278] Timeout 28: Found 1 model versions that have in-flight inferences

I1202 14:25:47.935836 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

I1202 14:25:48.936002 6685 server.cc:278] Timeout 27: Found 1 model versions that have in-flight inferences

I1202 14:25:48.936052 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

------------------------------ Captured log call -------------------------------

ERROR    traitlets:client.py:795 Timeout waiting for execute reply (180s).


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    163    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          624    356    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    108    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              182    125    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               772    345    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     74    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     23    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     54    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/pytorch.py                              31     18    42%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     17    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     15    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             29     18    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/pytorch.py              62     37    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/workflow.py             47     29    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 157     49    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     17    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143    120    16%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     20    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     93    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     27    16%


TOTAL                                                                                                 15451   7521    51%


I1202 14:25:49.936131 6685 server.cc:278] Timeout 26: Found 1 model versions that have in-flight inferences

I1202 14:25:49.936197 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

I1202 14:25:50.936278 6685 server.cc:278] Timeout 25: Found 1 model versions that have in-flight inferences

I1202 14:25:50.936320 6685 server.cc:282] Model 'ensemble_model' (version 1) has 1 in-flight inferences

=================== 2 failed, 1 passed in 387.43s (0:06:27) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins14665030450957315295.sh

@mikemckiernan
Add information about the Merlin DAG
d9a832e 
- Define the important terms of the DAG.
- Incorporate Karl's information.
- Karl's info about Operators and ColumnSelectors.
- Karl's info about Dataset.
@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #731 of commit d9a832eb7f0af70f8bb5fb9ef6fff9ea2de6c674, no merge conflicts.
Running as SYSTEM
Setting status of d9a832eb7f0af70f8bb5fb9ef6fff9ea2de6c674 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/670/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_merlin/workspace
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/731/*:refs/remotes/origin/pr/731/* # timeout=10
 > git rev-parse d9a832eb7f0af70f8bb5fb9ef6fff9ea2de6c674^{commit} # timeout=10
Checking out Revision d9a832eb7f0af70f8bb5fb9ef6fff9ea2de6c674 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d9a832eb7f0af70f8bb5fb9ef6fff9ea2de6c674 # timeout=10
Commit message: "Add information about the Merlin DAG"
 > git rev-list --no-walk 787e88c486208b30f600428cb64a4fcfb85d73e0 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins16965490315391545973.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.21,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.21,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1384580172'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-fj89bw6x
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-fj89bw6x
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 4f3d92dc259971eee2fee319713b0d7a08a57c54
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+24.g4f3d92d) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+24.g4f3d92d) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+24.g4f3d92d) (0.3.0+12.g78ecddd)
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 8.7 MB/s eta 0:00:00
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 131.0 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+24.g4f3d92d) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+24.g4f3d92d) (1.20.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (65.5.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 100.4 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+24.g4f3d92d) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+24.g4f3d92d-py3-none-any.whl size=100465 sha256=9e078e5c0f14e333863422c927d1bb8b05de23f6d5ce1a81b6c02a6fada8a326
  Stored in directory: /tmp/pip-ephem-wheel-cache-1mql4pq4/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+24.g4f3d92d treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-ezv62w2x
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-ezv62w2x
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit e08a72c9c59416a9000e62d25548eb08367fc3fa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+61.ge08a72c9) (0.3.0+12.g78ecddd)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+61.ge08a72c9-py3-none-any.whl size=367208 sha256=3a0aae87a5281ba0a19c44b9b62af230019e8be709d9de8a566a543e84781d24
  Stored in directory: /tmp/pip-ephem-wheel-cache-3yogmqqi/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+61.ge08a72c9
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-8dx6jitp
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-8dx6jitp
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 51af616069689b3ba57e8842a6f4a20377795df7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+13.g51af6160) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+13.g51af6160) (0.3.0+12.g78ecddd)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.55.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+13.g51af6160) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+13.g51af6160-cp38-cp38-linux_x86_64.whl size=257601 sha256=a278d3245efad13bbf6e17cf551af1503047723f11d8052f3c497707ea9f93e6
  Stored in directory: /tmp/pip-ephem-wheel-cache-7c0l8zg9/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=069f794919f7936ecdfb4760b329d0008c08356381b46e1cbdd3ef98fc2e0b65
  Stored in directory: /tmp/pip-ephem-wheel-cache-7c0l8zg9/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+13.g51af6160
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-hv7aoz_6
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-hv7aoz_6
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=770ff73574bc3bca20da4327719606791efede9e204773388b79d1d6898d3078
  Stored in directory: /tmp/pip-ephem-wheel-cache-yxigwa8y/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7fda08867d30>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7fda08867d30>, {'id': '1a270371', 'cell_type': 'code', 'metadata'...pass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}, 55)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7fda07cdff40>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...rver is already running on localhost:8001\nRuntimeError: Another tritonserver is already running on localhost:8001\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7fda08867d30>

cell = {'id': '1a270371', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:39:11.769018Z',...8;01mpass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}

cell_index = 55, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7fda08867d30>

cell = {'id': '1a270371', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-12-02T14:39:11.769018Z',...8;01mpass\x1b[39;00m\n', '\x1b[0;31mRuntimeError\x1b[0m: Another tritonserver is already running on localhost:8001']}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '228bb35c-e7ef-4f...e, 'engine': '228bb35c-e7ef-4f2f-ba8d-679d1748e288', 'started': '2022-12-02T14:39:11.769366Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    116�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E           �[1;32m    117�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E           �[1;32m    118�[0m �[38;5;124;03mresponse.�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    136�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E           �[1;32m    137�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    138�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[0;32m--> 139�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[1;32m    140�[0m     response �[38;5;241m=�[39m send_triton_request(

E           �[1;32m    141�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E           �[1;32m    142�[0m     )

E           �[1;32m    144�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E           �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E           �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E           �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E           �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:58�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E           �[1;32m     56�[0m     �[38;5;28;01mwith�[39;00m grpcclient�[38;5;241m.�[39mInferenceServerClient(grpc_url) �[38;5;28;01mas�[39;00m client:

E           �[1;32m     57�[0m         �[38;5;28;01mif�[39;00m client�[38;5;241m.�[39mis_server_ready():

E           �[0;32m---> 58�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mAnother tritonserver is already running on �[39m�[38;5;132;01m{�[39;00mgrpc_url�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m)

E           �[1;32m     59�[0m �[38;5;28;01mexcept�[39;00m tritonclient�[38;5;241m.�[39mutils�[38;5;241m.�[39mInferenceServerException:

E           �[1;32m     60�[0m     �[38;5;28;01mpass�[39;00m

E

E           �[0;31mRuntimeError�[0m: Another tritonserver is already running on localhost:8001

E           RuntimeError: Another tritonserver is already running on localhost:8001


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"





      tb2.inject(





            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )



tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:




../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7fda08867d30>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    116�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E               �[1;32m    117�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E               �[1;32m    118�[0m �[38;5;124;03mresponse.�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    136�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E               �[1;32m    137�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m    138�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[0;32m--> 139�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[1;32m    140�[0m     response �[38;5;241m=�[39m send_triton_request(

E               �[1;32m    141�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E               �[1;32m    142�[0m     )

E               �[1;32m    144�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E               �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E               �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E               �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E               �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E               �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:58�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E               �[1;32m     56�[0m     �[38;5;28;01mwith�[39;00m grpcclient�[38;5;241m.�[39mInferenceServerClient(grpc_url) �[38;5;28;01mas�[39;00m client:

E               �[1;32m     57�[0m         �[38;5;28;01mif�[39;00m client�[38;5;241m.�[39mis_server_ready():

E               �[0;32m---> 58�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mAnother tritonserver is already running on �[39m�[38;5;132;01m{�[39;00mgrpc_url�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m)

E               �[1;32m     59�[0m �[38;5;28;01mexcept�[39;00m tritonclient�[38;5;241m.�[39mutils�[38;5;241m.�[39mInferenceServerException:

E               �[1;32m     60�[0m     �[38;5;28;01mpass�[39;00m

E

E               �[0;31mRuntimeError�[0m: Another tritonserver is already running on localhost:8001

E               RuntimeError: Another tritonserver is already running on localhost:8001


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-02 14:37:54.380606: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:37:58.551492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:37:58.552270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:37:58.553047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:37:58.553650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 448 points to 32 centroids: please provide at least 1248 training points

2022-12-02 14:39:04.833965: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-12-02 14:39:08.933576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-12-02 14:39:08.934478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-12-02 14:39:08.935842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-12-02 14:39:08.936998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7fdae62686a0>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '5252563f-085e-4cb...e, 'engine': '5252563f-085e-4cb7-8c71-6b95171b5de5', 'started': '2022-12-02T14:39:20.749264Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-12-02 14:39:23,740 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:23,750 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:23,753 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:23,791 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:24,313 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:24,315 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:26,935 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-0cidvacx', purging

2022-12-02 14:39:26,936 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:27,206 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:27,207 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,090 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-3ie7iu6v', purging

2022-12-02 14:39:30,091 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:30,094 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:30,132 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:30,211 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:30,616 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,617 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,626 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,627 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,706 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,707 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,787 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:30,788 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:33,338 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-zbqqvu35', purging

2022-12-02 14:39:33,338 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-_4wr9vyl', purging

2022-12-02 14:39:33,338 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-vh0i952k', purging

2022-12-02 14:39:33,339 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-liuzrnwd', purging

2022-12-02 14:39:33,339 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-02 14:39:33,616 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-02 14:39:33,617 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 60 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       356    213    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                       99     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        55     31    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       141     29    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    163    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py                  127    100    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          347    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          624    357    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     17    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                117     58    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     175     95    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    111    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 242    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    161    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              182    125    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                106     71    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     84    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py                         9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                64     39    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    268    148    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               772    350    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              123     90    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               98     63    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     37    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       229     58    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py                               122     74    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               32      8    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     23    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               118     54    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/fil.py                                 211    117    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/pytorch.py                              31     18    42%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     17    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     15    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             29     18    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/fil.py                  95     66    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/pytorch.py              62     37    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           53      5    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/workflow.py             47     29    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 157     49    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              49     34    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                          143    120    16%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     45    38%


TOTAL                                                                                                 15237   7470    51%


=================== 2 failed, 1 passed in 118.83s (0:01:58) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins17929465269154037843.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@mikemckiernan mikemckiernan

Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
Merlin 22.12
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mikemckiernan @nvidia-merlin-bot @bschifferer