pyo3 crate recompiled on (every?) build #1708

cswinter · 2021-07-01T04:55:50Z

I don't have a minimal repro yet, but the pyo3 crate seems to be recompiled on most builds which significantly increases compilation times. I'm on pyo3 version 0.13.2, rustc version 1.53.0.

davidhewitt · 2021-07-01T06:30:24Z

See also #1551

This might be fixed on main / upcoming 0.14 release depending on your build environment.

I think that rust-analyzer runs things like cargo check which can invalidate pyo3's build config unless the environments carefully match.

This is worthy of a topic in the guide, and probably there could be more discussion whether we need to make further changes.

cswinter · 2021-07-01T16:12:09Z

Ah good to know! I tried setting PYO3_PYTHON both when building manually and in rust-analyzer.server.extraEnv but that doesn't seem to have done the trick yet. I think I'll just remove PyO3 completely from Rust builds for now which I have mostly set up that way anyway already because of that linker issue. Happy to do some more testing/debugging on this once 0.14 lands.

davidhewitt · 2021-07-02T05:53:29Z

Thanks. I've also been considering reaching out to the rust-analyzer folks who might be more familiar with all the cases when pyo3's build.rs is invoked, however haven't gotten around to that yet.

mejrs · 2021-07-02T07:04:51Z

I have pyo3 as an optional dependency for this reason.

dbr · 2021-09-03T14:44:30Z

I have this happening semi-randomly (happens "almost all the time", but, not always). I am running rust-analyzer (in emacs via lsp-mode), and mostly notice this when switching to terminal to cargo run. It was mysterious how rust-analyzer was causing the command-line builds to change, so I looked into it a bit:

Running build with

CARGO_LOG=cargo::core::compiler::fingerprint=trace cargo build

I get the following output (can probably change 'trace' to 'info')

INFO  cargo::core::compiler::fingerprint]     err: env var `PATH` changed: previously Some("/home/dbr/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"), now Some("/home/dbr/.cargo/bin:/home/dbr/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games")

Basically ~/.cargo/bin is in $PATH twice in rust-analyzer but only once in regular build (I assume, might be other way around), which then triggers a re-build of PyO3 as intended (as per the #1551 issue David mentioned above)

rust-analyzer uses the same target/ directory, which is how it can interfere with running cargo build. Specifically, the files like target/debug/.fingerprint/pyo3-a2b72c2ee6b7ccd3 stores the previous value of $PATH like so:

[..snip..] {"RerunIfEnvChanged":{"var":"PATH","val":"/home/dbr/.cargo/bin:/home/dbr/.cargo/bin:[..snip..]"}}],"rustflags":[],"metadata":0,"config":0,"compile_kind":0} [..snip..]

So I guess this issue basically becomes: why is ~/.cargo/bin getting added to $PATH twice (which is most certainly an issue for either rust-analyzer, or possibly cargo?)

dbr · 2021-09-14T09:37:41Z

After much poking around, I think the problem lies in rustup (not rust-analyzer nor cargo), so have filed a bug report in that project:

rust-lang/rustup#2848

For now I think configuring CARGO_TARGET_DIR for rust-analyzer would seem to be the most reliable workaround (at the cost of having two separate builds)

davidhewitt · 2021-09-16T23:02:03Z

Thanks for the investigation! If the rustup bug getting fixed does indeed remove this frequently-encountered papercut, I'll be very happy 😄

…build-config rebuilds See PyO3/pyo3#1708 (comment)

rbozan · 2023-01-20T21:03:44Z

The PR that fixes it in rustup (rust-lang/rustup#2849) is merged

xnerhu · 2023-02-06T19:32:26Z

The PR that fixes it in rustup (rust-lang/rustup#2849) is merged

When I run yarn build from Nodejs where in package.json I have scripts

"scripts": {
   "build": "cargo build"
}

It recompiles pyo3 every time. But when I run cargo build directly in the terminal, it doesn't rebuild pyo3.

windows 11
pyo3 0.18.0
rustup 1.25.2 (17db695f1 2023-02-01)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.67.0 (fc594f156 2023-01-24)`

I think it's because yarn adds temp path, which changes every time. Tho running via npm run build doesn't work too.

rustup\toolchains\stable-x86_64-pc-windows-msvc\bin;C:\Users\xnerhu\AppData\Local\Temp\yarn--1675712873242-0.

bazhenov · 2023-02-10T11:40:22Z

Unfortunately, problem still remains. In my case it is only manifest itself when rust-analyzer is running in IDE.

antimora · 2023-02-15T03:34:49Z

I can confirm too. I use rust-analyzer (extension v0.3.1402) with vscode and the py03 gets recompiled every time.

use pyo3::{types::PyDict, types::PyModule, PyResult, Python};

   Compiling pyo3-build-config v0.18.0
   Compiling pyo3-ffi v0.18.0
   Compiling pyo3 v0.18.0

davidhewitt · 2023-02-15T07:49:00Z

Can you try running your builds with CARGO_LOG=cargo::core::compiler::fingerprint=info as suggested above, and show here what is causing the cache invalidation?

bazhenov · 2023-02-15T13:28:59Z

I managed to solve a problem. Or at least found a decent workaround.

Can you try running your builds with CARGO_LOG=cargo::core::compiler::fingerprint=info as suggested above, and show here what is causing the cache invalidation?

Seems like the same issue: err: env var 'PATH' changed.

Full log:

[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] dependency on `build_script_build` is newer than we are 1676466118.936326620s > 1676466053.631031496s "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1"
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] dependency on `build_script_build` is newer than we are 1676466119.140438444s > 1676466056.897465668s "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1"
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for crab v0.1.0 (/Users/bazhenov/Developer/crab)/Build/TargetInner { ..: lib_target("crab", ["lib"], "/Users/bazhenov/Developer/crab/src/lib.rs", Edition2021) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/Build/TargetInner { ..: lib_target("pyo3", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/b73abd80d1b5c6e5) != old (build_script_build/7d829398a44f12b4)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/Build/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (pyo3_build_config/1743a4d849f4181b) != old (pyo3_build_config/b34bbfa27ea017eb)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-build-config v0.18.1/Build/TargetInner { ..: lib_target("pyo3-build-config", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-build-config-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/5fe9711f85c67c02) != old (build_script_build/d8fecd9d3b0d8efe)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-build-config v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-build-config-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: env var `PATH` changed: previously Some("/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/bazhenov/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin:/usr/local/opt/fzf/bin"), now Some("/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/sbin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin:/usr/local/opt/fzf/bin:/usr/local/sbin:/usr/local/bin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin")
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/d41ac0404f890cd7) != old (build_script_build/9b9073f7bf4231be)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/Build/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (pyo3_build_config/1743a4d849f4181b) != old (pyo3_build_config/b34bbfa27ea017eb)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/Build/TargetInner { ..: lib_target("pyo3-ffi", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for crab v0.1.0 (/Users/bazhenov/Developer/crab)/Build/TargetInner { name: "crab", doc: true, ..: with_path("/Users/bazhenov/Developer/crab/src/main.rs", Edition2021) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
   Compiling pyo3-build-config v0.18.1
   Compiling pyo3-ffi v0.18.1
   Compiling pyo3 v0.18.1
   Compiling crab v0.1.0 (/Users/bazhenov/Developer/crab)
    Finished dev [unoptimized + debuginfo] target(s) in 7.76s

Looking at src/impl_.rs in the pyo3-build-config I realize that cargo:rerun-if-env-changed=PATH directive is applied only if no PYO3_PYTHON environment variable is provided. So I've added following configuration to .cargo/config.toml

[env]
PYO3_PYTHON = "/usr/local/bin/python3"

And now everything works like a charm 🍾. Hope this helps.

max-sixty · 2023-02-15T17:27:54Z

Thanks for the repro and suggestion @bazhenov !

[env]
PYO3_PYTHON = "/usr/local/bin/python3"

Would there be an approach that generalizes to others who have different locations for their Python executable?

bazhenov · 2023-02-16T10:42:14Z

Would there be an approach that generalizes to others who have different locations for their Python executable?

It's kinda is. Cargo config can be located not in project but home directory – https://doc.rust-lang.org/cargo/reference/config.html

alex · 2023-04-01T20:09:29Z

We see:

     Dirty pyo3-build-config v0.18.2: the env variable PYO3_PYTHON changed

on basically every CI build. I believe I can explain what's happening:

We use pyo3 + setuptools-rust. In CI we do pip install . and then use CARGO_TARGET_DIR to ensure we have a consistent target/ dir so we can cache already built Rust dependencies between builds.

Because we are a PEP517 package, when pip install . is executed, pip will create an sdist for the package, create a temporary virtualenv, unpack the sdist, and then install the package there. The python that is used to invoke setuptools (and setuptools-rust) which in turn shells out to cargo rustc is the python from the temporary virtualenv, and therefore changes on every invocation.

This leads to invalidating the PYO3_PYTHON env var on every single build.

alex · 2023-04-01T20:45:16Z

Disregard this, it was entirely wrong 😭. Turns out we were sharing caches between different tox (or nox) jobs (e.g. lint vs. docs) and the environment name is in the path.

tadeohepperle · 2024-04-08T08:25:56Z

Is there any update on this? I still experience PyO3 recompiling every time when using rust-analyzer.

Jgfrausing · 2024-05-13T06:39:04Z

They seem to have found a workaround. The documentation in ipetkov/crane#486 specifies setting a path variable for the python binary in NIX. It can also just be put into the [env] section of .cargo/config.toml

...

[env]
PYO3_PYTHON = "path/to/python"

wyfo · 2024-05-13T06:53:48Z

@Jgfrausing This workaround was already described in a previous comment.
It seems someone needs to open a PR to update the FAQ, I will do it immediately.

Jgfrausing · 2024-05-13T07:22:22Z

Thanks. Maybe also mark this as solved, so that others know to look for a solution within this issue.

EDIT: I see, you've also covered that. Thanks, mate.

tadeohepperle · 2024-05-16T10:14:35Z

I tried the workaround with

[env]
PYO3_PYTHON = "./venv/bin/python"

but it does not change anything for me. When rust-analyzer is active, every code edit I make, results in a complete (~20 sec) rebuild of pyo3-macros and other dependencies. :(

henryiii · 2024-05-22T16:35:45Z

# .cargo/config.toml
[env]
PYO3_PYTHON = "./.venv/bin/python"

Worked for me. Had to make it (uv venv) and vscode asked if I wanted to use it, I said yes. I also manually activated it in the existing terminal pane (new ones are fine, too, and it does add a little "request to relaunch" warning in the sidebar). Reran both cargo test in the terminal and saved the file, and now it's not recompiling PyO3 every time.

metasim · 2024-05-25T17:51:56Z

FWIW, I'm having the same problem when using RustRover. Thanks to everyone for the workaround suggestions.

davidhewitt mentioned this issue Jul 10, 2021

Cache the Rust build PyO3/setuptools-rust#152

Closed

davidhewitt added bug documentation labels Aug 26, 2021

dbr mentioned this issue Sep 14, 2021

Deduplicate PATH entries added during build (e.g. ~/.cargo/bin) rust-lang/rustup#2848

Closed

dmartin mentioned this issue Feb 7, 2022

Fix duplicated PATH entries rust-lang/rustup#2849

Merged

davidhewitt mentioned this issue Feb 9, 2022

Maturin develop always rebuilds every dependency PyO3/maturin#504

Closed

davidhewitt mentioned this issue Apr 15, 2022

Unnecessary rebuilds due to ineffective path changes #2308

Closed

messense mentioned this issue Apr 25, 2022

Wheel is rebuilt every time, even if there are no changes to the binary PyO3/maturin#884

Closed

2 tasks

sapir added a commit to sapir/binq that referenced this issue Nov 23, 2022

Configure CARGO_TARGET_DIR for rust-analyzer to fix unnecessary pyo3-…

66c88bd

…build-config rebuilds See PyO3/pyo3#1708 (comment)

vlaci mentioned this issue Dec 18, 2023

wants to rebuild pyo3 ipetkov/crane#414

Closed

This was referenced Apr 21, 2024

Polars: bounded-DP mean via postprocessing opendp/opendp#890

Merged

PyO3 recompiles on every build opendp/opendp#1537

Open

wyfo linked a pull request May 13, 2024 that will close this issue

docs: Add a FAQ section about recompilation due to IDE #4183

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyo3 crate recompiled on (every?) build #1708

pyo3 crate recompiled on (every?) build #1708

cswinter commented Jul 1, 2021

davidhewitt commented Jul 1, 2021

cswinter commented Jul 1, 2021

davidhewitt commented Jul 2, 2021

mejrs commented Jul 2, 2021

dbr commented Sep 3, 2021 •

edited by birkenfeld

dbr commented Sep 14, 2021

davidhewitt commented Sep 16, 2021

rbozan commented Jan 20, 2023

xnerhu commented Feb 6, 2023 •

edited

bazhenov commented Feb 10, 2023

antimora commented Feb 15, 2023

davidhewitt commented Feb 15, 2023

bazhenov commented Feb 15, 2023

max-sixty commented Feb 15, 2023

bazhenov commented Feb 16, 2023

alex commented Apr 1, 2023

alex commented Apr 1, 2023

tadeohepperle commented Apr 8, 2024

Jgfrausing commented May 13, 2024

wyfo commented May 13, 2024

Jgfrausing commented May 13, 2024 •

edited

tadeohepperle commented May 16, 2024

henryiii commented May 22, 2024

metasim commented May 25, 2024

pyo3 crate recompiled on (every?) build #1708

pyo3 crate recompiled on (every?) build #1708

Comments

cswinter commented Jul 1, 2021

davidhewitt commented Jul 1, 2021

cswinter commented Jul 1, 2021

davidhewitt commented Jul 2, 2021

mejrs commented Jul 2, 2021

dbr commented Sep 3, 2021 • edited by birkenfeld

dbr commented Sep 14, 2021

davidhewitt commented Sep 16, 2021

rbozan commented Jan 20, 2023

xnerhu commented Feb 6, 2023 • edited

bazhenov commented Feb 10, 2023

antimora commented Feb 15, 2023

davidhewitt commented Feb 15, 2023

bazhenov commented Feb 15, 2023

max-sixty commented Feb 15, 2023

bazhenov commented Feb 16, 2023

alex commented Apr 1, 2023

alex commented Apr 1, 2023

tadeohepperle commented Apr 8, 2024

Jgfrausing commented May 13, 2024

wyfo commented May 13, 2024

Jgfrausing commented May 13, 2024 • edited

tadeohepperle commented May 16, 2024

henryiii commented May 22, 2024

metasim commented May 25, 2024

dbr commented Sep 3, 2021 •

edited by birkenfeld

xnerhu commented Feb 6, 2023 •

edited

Jgfrausing commented May 13, 2024 •

edited