[BUG] azureml-sdk[databricks]==1.0.8 failed to install on Azure Databricks Runtime version 11.3 ML LTS

[lordaouy]

Description

azureml-sdk[databricks]==1.0.8 failed to be installed on Azure Databricks runtime version 11.3 ML LTS with the error:

Library installation attempted on the driver node of cluster 1009-112226-olf4iwj9 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, azureml-sdk[databricks]==1.0.8, --disable-pip-version-check) exited with code 1.     ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-9fji_eh0
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py", line 211, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py", line 184, in _package_data
        data = literal_eval("".join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py", line 156, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_c901922d59484a439661a520a5369f5f/setup.py", line 95, in _convert
        if isinstance(node, Str):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/77/19/c225d7dd6b3678e5f8b76b8101dc903a0f1799b7182eeab4d20b07a32878/ruamel.yaml-0.15.51.tar.gz#sha256=97eff81b23c73a2ea182c04a71e2aa939f3ca493b9b21e4ccbf52eeb000a3a46 (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-u2u9iy9v
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py", line 211, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py", line 184, in _package_data
        data = literal_eval("".join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py", line 156, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2222b40993d84a10bf24321a423d3753/setup.py", line 95, in _convert
        if isinstance(node, Str):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/df/65/605ac0e0c83c7e251d7cf3e8f393d76529a3acc4fac7deb2dfb6bab2799d/ruamel.yaml-0.15.50.tar.gz#sha256=5828c02a4a9d85bbb730ecb949d6775bb4581252e55de81abb91a20e49d83e19 (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-5_geaggj
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py", line 211, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py", line 184, in _package_data
        data = literal_eval("".join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py", line 156, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_33e235124fa040f7bf1a2eb55d35a574/setup.py", line 95, in _convert
        if isinstance(node, Str):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/7b/2e/be578ee125f006f6fa93a07e3330e9c12c8e197800e8e771af202a87876d/ruamel.yaml-0.15.49.tar.gz#sha256=6c8d082bc22adeb65ba2123c17cd399a59e3b1e1004ed3f52969d0dcc6be4e39 (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_6zblvk4
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py", line 211, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py", line 184, in _package_data
        data = literal_eval("".join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py", line 156, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_34e1d342a3734067a563eecfc51b0a8b/setup.py", line 95, in _convert
        if isinstance(node, Str):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a7/68/2429de61a7fcba5bda68c3aeaa618205a0fc5e4fe8678be9f347409b0f1f/ruamel.yaml-0.15.48.tar.gz#sha256=15f203230c5100bf361b954a49cf28961fc71a9a2dbcab7c6494ad2ab806eed4 (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-jan7kme6
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py", line 186, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py", line 164, in _package_data
        data = literal_eval(u''.join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py", line 136, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_01493d6cbf6640ffb1ff2885616b93fe/setup.py", line 83, in _convert
        if isinstance(node, (Str, Bytes)):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d0/5d/a4ef414c7ff9a0206c734894b5e525daec580046ac9dd80e9c31adfd96f0/ruamel.yaml-0.15.47.tar.gz#sha256=4b9de78a8b54ce0fcc04cdddeba16cd506f879189efc15ec865061ea7c93843a (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-c8gj6gn9
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py", line 186, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py", line 164, in _package_data
        data = literal_eval(u''.join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py", line 136, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_71ebce3601ed4cd88383874d0488e75e/setup.py", line 83, in _convert
        if isinstance(node, (Str, Bytes)):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/3d/46/a08f44a2a919e32689dca849ebfcb4f71f5e91e18f840bd49a88dc157a14/ruamel.yaml-0.15.46.tar.gz#sha256=8f048085a58ca59353c2c283e5f14af387ab6a1a7ae5d6ec26056bc2e7a396f0 (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-qjwmiblm
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py", line 186, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py", line 164, in _package_data
        data = literal_eval(u''.join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py", line 136, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_d033376417b14c24912318212b1ab201/setup.py", line 83, in _convert
        if isinstance(node, (Str, Bytes)):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/63/a5/dba37230d6cf51f4cc19a486faf0f06871d9e87d25df0171b3225d20fc68/ruamel.yaml-0.15.45.tar.gz#sha256=096691b0958514da21d19ae40255569f027b5b90530c55faf1d74ff16b2f256b (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-mnxqqf6h
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py", line 186, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py", line 164, in _package_data
        data = literal_eval(u''.join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py", line 136, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_66f432a79514407199a8bb8a6ab7b8c1/setup.py", line 83, in _convert
        if isinstance(node, (Str, Bytes)):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cd/8f/cce2403b82c6939ada5fd39488f0ce91729581e2d17436cba1c2020d2997/ruamel.yaml-0.15.44.tar.gz#sha256=9c70302a81b51f2ed3b6a42abb9128c9a9beaa765b73ff317310eb59c4a5131a (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1:
     command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r3u02zfj
         cwd: /tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py", line 186, in <module>
        pkg_data = _package_data(__file__.replace('setup.py', '__init__.py'))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py", line 164, in _package_data
        data = literal_eval(u''.join(lines))
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py", line 136, in literal_eval
        return _convert(node_or_string)
      File "/tmp/pip-install-vrmfzl99/ruamel-yaml_2a77f9fba7574ed5b467e5e1384369d3/setup.py", line 83, in _convert
        if isinstance(node, (Str, Bytes)):
    NameError: name 'Str' is not defined
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/20/bc/202519c9242e4923d4a12b08be439e66fbcea541dbed5ae9a36d0b9cfca2/ruamel.yaml-0.15.43.tar.gz#sha256=4e3672846ea0ff1ea97ee2004075b108b2dc5deecaab30a93cef0c751325befd (from https://pypi.org/simple/ruamel-yaml/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Cannot install azure-cli-core==2.53.0, azureml-core, azureml-sdk and azureml-sdk[databricks]==1.0.8 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

In which platform does it happen?

Azure Databricks 11.3 LTS ML (includes Apache Spark 3.3.0, Scala 2.12)

How do we replicate the issue?

Install it in a cluster with Databricks runtime version 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12).

Expected behavior (i.e. solution)

Other Comments

[miguelgfierro]

this should be fixed with the new release 1.2.0