ludwig datasets download throws errors

[AnoopRKulkarni]

Dear all,

I am using Ubuntu 22.04. I tried with both Python v 3.8.5 and 3.10.2. I installed them with pyenv and created virtualenv using pyenv and then did "pip install ludwig". The installation goes through smoothly, but when I run datasets download command I get the same error irrespective of the machine (laptop/desktop) or Python version.

Do advise what could be the issue here. Here is the trace..

ludwig v0.6.4 - Datasets download

Traceback (most recent call last):
File "/home/anoop/.pyenv/versions/ludwig-ai/bin/ludwig", line 8, in
sys.exit(main())
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/cli.py", line 172, in main
CLI()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/cli.py", line 67, in init
getattr(self, args.command)()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/cli.py", line 167, in datasets
datasets.cli(sys.argv[2:])
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/datasets/init.py", line 83, in cli
download_dataset(args.dataset, args.output_dir)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/datasets/init.py", line 34, in download_dataset
dataset_obj.process()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/datasets/base_dataset.py", line 53, in process
self.download()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/datasets/base_dataset.py", line 48, in download
self.download_raw_dataset()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/ludwig/datasets/mixins/download.py", line 160, in download_raw_dataset
fs.get(url, os.path.join(tmpdir, filename), recursive=True)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 114, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 99, in sync
raise return_result
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 54, in _runner
result[0] = await coro
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 531, in _get
rpaths = await self._expand_path(rpath, recursive=recursive)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 737, in _expand_path
out = await self._expand_path([path], recursive, maxdepth)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.8/site-packages/fsspec/asyn.py", line 761, in _expand_path
raise FileNotFoundError(path)
FileNotFoundError: ['https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv']

Many thanks and regards

~anoop

[tgaddair]

Thanks for reporting @AnoopRKulkarni! @connor-mccorm can you take a look?

[connor-mccorm]

Absolutely! Will get back here when I have some more info ASAP!

[connor-mccorm]

Hi @AnoopRKulkarni, could you possibly show me the command that you are using to load this dataset? I'm trying to repro on my end and am wondering which code path to work through. Additionally, could you try running this command and see if it produces the same error:

import ludwig.datasets as datasets
train_df, validation_df, test_df = datasets.agnews.load()

Let me know, thanks!

[AnoopRKulkarni]

Apologies for not including the command I was trying for loading datasets in my OP. It was command line

$ ludwig datasets download agnews

Also, I tried your other commands with Python.. and I get the same error as command line.

Thanks
~anoop

[connor-mccorm]

No worries at all! I will look into this and see if I can repro, then will work on a solution. I will keep you updated with progress as it comes!

[AnoopRKulkarni]

Thanks a bunch!

Also, another data point if it helps.. If I execute the same from the Colab notebook, it works! I get no errors. So guess it has to do with Ubuntu 22.04 installation?

I will keep you informed, if I find anything new as well.

regards
~anoop

[connor-mccorm]

Of course! And yes, the environment setup was definitely my top suspicion. Definitely keep me posted and I will do the same!

[AnoopRKulkarni]

Thanks..

If I do it through Python, get some SSL warning that leads to the eventual command line error.. here is the trace..

import ludwig.datasets as datasets
train_df, val_df, test_df = datasets.agnews.load()
train.csv: 0.00B [00:00, ?B/s]
Failed to download dataset

Traceback (most recent call last):
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 261, in _download_and_process
self.download(kaggle_username=kaggle_username, kaggle_key=kaggle_key)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 310, in download
urllib.request.urlretrieve(url, downloaded_file_path, t.update_to)
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 241, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/home/anoop/.pyenv/versions/3.10.2/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Traceback (most recent call last):
File "", line 1, in
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 287, in load
self._download_and_process(kaggle_username=kaggle_username, kaggle_key=kaggle_key)
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 264, in _download_and_process
self.verify()
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 315, in verify
digest = _sha256_digest(os.path.join(self.raw_dataset_dir, filename))
File "/home/anoop/.pyenv/versions/ludwig-ai/lib/python3.10/site-packages/ludwig/datasets/loaders/dataset_loader.py", line 90, in _sha256_digest
with open(file_path, "rb", buffering=0) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/anoop/.ludwig_cache/agnews_1.0/raw/test.csv'

Could that be helpful in locating the issue?

~anoop

[AnoopRKulkarni]

OK. it is solved.

I checked for certificate related errors and needed to run the following on my Ubuntu.

#sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

Ludwig is now running smoothly.

Apologies for diverting your attention for an issue that maybe was just local.

Really appreciate the interaction.

Thanks and regards
~anoop

[connor-mccorm]

This is awesome! No worries, I'm just glad you got it figured out:) Please feel free to continue the discussion or raise an issue if you encounter any other troubles!