python3Packages.torch: drop submodules in favor of Nixpkgs #239291
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ConnorBaker
force-pushed
the
feat/torch-use-system-libs
branch
2 times, most recently
from
7a1b72b
to
3513293
Compare
ConnorBaker
added a commit
to ConnorBaker/pytorch
that referenced
this pull request
Jul 5, 2023
Fixes pytorch#44255. This is part of larger work I'm doing to allow for more `USE_SYSTEM_*` options to allow Nix to have faster re-builds of PyTorch: NixOS/nixpkgs#239291. Pull Request resolved: pytorch#104611 Approved by: https://github.com/ezyang, https://github.com/malfet
ConnorBaker
force-pushed
the
feat/torch-use-system-libs
branch
from
864c0e9
to
448fb88
Compare
|
@Dessix as you had asked about potential ways to help with this: The near-term goal is to be able to build Longer-term, I'd like to be able to build If you wanted to take a look into the targets CMake supports building, and whether we're able to build libtorch_cpu and libtorch_cuda independently of each other, that would be helpful! Alternatively, if you have any CMake experience, I'd love a second eye on the CMake PRs I've submitted upstream. |
ConnorBaker
force-pushed
the
feat/torch-use-system-libs
branch
from
aaeaf3b
to
09ea472
Compare
ofborg
bot
added
10.rebuild-darwin: 1-10
10.rebuild-linux: 11-100
and removed
10.rebuild-darwin: 11-100
10.rebuild-linux: 101-500
labels
|
This one can definitely be split into its own PR: 98281d7 |
This change which involves creating multiple outputs for CUDA redistributable packages. We use a script to find out, ahead of time, the outputs each redist package provides. From that, we are able to create multiple outputs for supported redist packages, allowing users to specify exactly which components they require. Beyond the script which finds outputs ahead of time, there is some custom code involved in making this happen. For example, the way Nixpkgs typically handles multiple outputs involves making `dev` the default output when available, and adding `out` to `dev`'s `propagatedBuildInputs`. Instead, we make each output independent of the others. If a user wants only to include the headers found in a redist package, they can do so by choosing the `dev` output. If they want to include dynamic libraries, they can do so by specifying the `lib` output, or `static` for static libraries. To avoid breakages, we continue to provide the `out` output, which becomes the union of all other outputs, effectively making the split outputs opt-in.
12 tasks
ConnorBaker
force-pushed
the
feat/torch-use-system-libs
branch
from
225a72a
to
faaf22c
Compare
…ng them to buildInputs
wegank
added
2.status: stale
stale
bot
removed
the
2.status: stale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Closes #239211.
Todo:
USE_ITTflag Intel profiling: https://github.com/intel/ittapi)PyTorch efforts regarding upstream CMake:
FindCUDA.cmakepytorch/pytorch#76082Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)