Installed package missing files

[austinmw]

Hi, I have the following in my setup.cfg:

[options.extras_require]
sherlock =
     sherlock @ git+https://github.com/mitmedialab/sherlock-project.git
     tensorflow
     gensim
     nltk
     gdown

Which installs this package when I do pip install package_name[sherlock]. However I noticed that several of the larger files in this repository are missing after the install (for example, par_vec_trained_400.pkl). How do I modify the setup so that it downloads everything from the repository?

Do I need to install it in editable mode somehow?

[FlorianWilhelm]

Hi @austinmw, the documentation of sherlock says that you have to call the download_data() function in the helpers module to download those large files. So this is intended behaviour of sherlock. You cannot really change this as

pip install git+https://github.com/mitmedialab/sherlock-project.git

would also not download the large files and this is basically what you do by specifying [options.extras_require].

[austinmw]

Hi @FlorianWilhelm, thanks for your response! The download_data() function downloads the data files from Google Drive, but that's not what I'm looking for. I want the model files, which are not in Google Drive, but located directly in the repository (you would get them if you did a git clone).

[abravalheri]

@austinmw the sherlock package is not configured to include these files.

If you have a look on their setup.py, you can see that it does not contain any configuration for package_data or include_package_data. This means that all non-Python files are simply ignored when distributing the package.

In the end of the day, the authors of sherlock either intentionally or accidentally are skipping these files in the distribution installed by pip.

You can verify this behaviour yourself by running the following commands:

git clone https://github.com/mitmedialab/sherlock-project.git
cd sherlock-project
pipx run build # you need `pipx` installed
unzip -l dist/sherlock-*.whl

[austinmw]

Thanks, didn't know what was why the non-Python files are ignored. Makes sense!

[abravalheri]

Yeah, it is a bit tricky.

That is why in the PyScaffold template we use setuptools-scm and include_package_data=True. It does the trick of adding all the files in the repository to the distribution.

[FlorianWilhelm]

@austinmw, maybe you can convince the authors of sherlock to migrate their project to PyScaffold ;-)