CROSS_CONTAINER_OPTS does not replace --platform linux/amd64

[leet4tari]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

I don't think that the CROSS_CONTAINER_OPTS env is replacing the --platform linux/amd64 command, at least not on Apple Silicon.

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu, x86_64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.5 (19be834 2024-05-17)

Example

Running

export CROSS_CONTAINER_OPTS="--platform linux/arm64"
export CROSS_BUILD_OPTS="--platform linux/arm64"
export CROSS_DEBUG=1

then trying run

cross build \
  --target aarch64-unknown-linux-gnu \
  --bin minotari_miner -v
[src/cargo.rs:156] ->
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
[src/rustc.rs:370] ->
+ rustc --print sysroot
[src/docker/engine.rs:220] ->
+ /usr/local/bin/docker
[src/docker/engine.rs:295] ->
+ /usr/local/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
[src/docker/engine.rs:163] ->
+ /usr/local/bin/docker info -f {{.SecurityOptions}}
[src/rustup.rs:89] ->
+ rustup toolchain list
[src/rustup.rs:115] ->
+ rustup target list --toolchain nightly-2024-03-01-x86_64-unknown-linux-gnu
[src/rustup.rs:242] ->
+ rustup component list --toolchain nightly-2024-03-01-x86_64-unknown-linux-gnu
[src/docker/custom.rs:177] ->
+ /usr/local/bin/docker buildx build --platform linux/amd64 --progress auto --label 'org.cross-rs.for-cross-target=aarch64-unknown-linux-gnu' --label 'org.cross-rs.runs-with=x86_64-unknown-linux-gnu' --label 'org.cross-rs.workspace_root=/Users/leet/src/tari/tari-projects/tari' --tag localhost/cross-rs/cross-custom-tari:aarch64-unknown-linux-gnu-6c732-pre-build --build-arg 'CROSS_SCRIPT=./scripts/cross_compile_hack.sh' --build-arg 'CROSS_TARGET=aarch64-unknown-linux-gnu' --build-arg 'CROSS_DEB_ARCH=arm64' --file /Users/leet/src/tari/tari-projects/tari/target/aarch64-unknown-linux-gnu/Dockerfile.aarch64-unknown-linux-gnu-custom --platform linux/arm64 --output 'type=docker' /Users/leet/src/tari/tari-projects/tari

You might notice docker buildx build --platform linux/amd64 with --platform linux/arm64 much later in the command, which I think force running x86 images, even thou I would like to use an aarch64/arm64 image.

Reason been, that I was looking to run a CPU native docker image as my base to cross-compile, I was expecting/hoping to use aarch64-unknown-linux-gnu image (Ubuntu 18.04), but kept running into rosetta errors.

Additional information / notes

No response

[Emilgardis]

to use a custom image with another platform you need to specify the platforms it can/should run with. See https://github.com/cross-rs/cross/blob/main/docs/config_file.md#targettargetimage

[target.aarch64-unknown-linux-gnu]
image.name = "alpine:edge"
image.toolchain = ["x86_64-unknown-linux-musl", "linux/arm64=aarch64-unknown-linux-musl"]

so in this case, you want

[target.aarch64-unknown-linux-gnu]
image.name = "my-image"
image.toolchain = ["aarch64-unknown-linux-gnu"]

[leet4tari]

Thanks for the reply @Emilgardis.

I had tried quite a few option and I might not have listed trying this:

[target.aarch64-unknown-linux-gnu]
image = "ubuntu:18.04"
pre-build = "./scripts/cross_compile_hack.sh"

[target.aarch64-unknown-linux-gnu.env]
passthrough = [
  "CARGO_BUILD_TARGET",
  "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc",
  "BINDGEN_EXTRA_CLANG_ARGS=--sysroot /usr/aarch64-linux-gnu/include/",
]

and my script does a uname -m which is listing x86 and not aarch64 as I had expected. Basic idea was that on MacOS, could use cross-rs and docker to cross-compile Linux binaries, with minimal fuss and effect.

The problem I am seeing, is that cross-rs is not letting me choose the docker platform to run/build on, even if I try and overload with either of the two env OPTS.

[Emilgardis]

You need to tell cross that the image can run on aarch64 platform, we assume x86_64 if nothing is specified.

To choose the platform, you need to add it as a valid platform for that image

[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
image.toolchain = ["aarch64-unknown-linux-gnu"]

There is however no way to tell cross "of all the possible platforms, please use x", we should probably make that possible

[leet4tari]

Thanks again @Emilgardis , can we not use the env CROSS_CONTAINER_OPTS to force docker to use a platform? That is something I saw suggested in an issue while looking through this.

What I was raising is that the docker run command, thou it does add the env to the run but at the end, does not replace what looks like the default of docker buildx build --platform linux/amd64 vs replacing the --platform part if put into the env.

[leet4tari]

What my original expectation was that the docker would be running with local CPU arch, unless overridden or forced, might that be better user expectation?

[Emilgardis]

To not have cross insert --platform if it's already been specified with CROSS_CONTAINER_OPTS you'd have to modify

cross/src/docker/image.rs

Line 437 in 19be834

pub fn specify_platform(&self, engine: &Engine, cmd: &mut std::process::Command) {

I think we can do that, it's consistent with what we do for --load on CROSS_BUILD_OPTS. There might however be other issues with that due to the expectation in the code that we've determined the platform and are using it.

What my original expectation was that the docker would be running with local CPU arch, unless overridden or forced, might that be better user expectation?

That's an interesting insight, it might be worth doing. My only concern is how moby/buildx now attests image provenence with unknown/unknown, which together with the issue presented in #1214 makes it kinda confusing.

[leet4tari]

That is the issue where I got the idea to try and force the docker platform, when I was able to debug that the arm64 was defaulting to using the x86_64 docker engine.

The idea I was raising and really hoping would be easy to fix or even test, would be to stop the defaulting to x86_64 or be able to use the --platform override, but my rust is not good enough to come up with sort of patch. Happy to try and test anything, if you or anybody else has some suggestions,.

Interestingly, somebody might want to force building with qemu or rosetta that is not the default docker target, but that is a different problem, I would just like to be able to use the native CPU arch, but been able to target a different OS, ie osx -> linux.

As for the docker images having unknown/unknown, I still don't understand this behaviour and have disable for now in my own projects.

[Emilgardis]

It's unclear to me if you've been able to solve your problem, the way to immediately solve your issue of not being able to specify --platform=linux/arm64 is the config I shared earlier, does that work for now?

With that config it's even possible to specify precisely what platform to use

[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
# syntax = <platform>=<toolchain>
image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

[leet4tari]

I missed your suggestion on how to possible target the platform, let me test and report back. Thanks @Emilgardis for all the support.

[leet4tari]

Thanks @Emilgardis , I was able to come right. I can cross-compile our project on both x86_64 and aarch64 OSX computers to Linux x86_64 and aarch64.

I am not sure I would have tried your suggestion from the documentation, thou that might just be me trying to get something do in a very specific kind of way.

Still not sure that it is right, but it works.

The docker command line might still be a bug, as it does not only issue one set of args as overrides, just duplicates what you have put into the env.

[leet4tari]

Adding some info and possible see which way this goes

[target.aarch64-unknown-linux-gnu]
image = "ubuntu:18.04"
# Need to use native aarch64 on Apple silicon
#image.name = "ubuntu:18.04"
#image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]
pre-build = "./scripts/cross_compile_ubuntu_18-pre-build.sh"

[target.x86_64-unknown-linux-gnu]
image = "ubuntu:18.04"
pre-build = "./scripts/cross_compile_ubuntu_18-pre-build.sh"

maybe better for me to open another issue regarding how the Apple Silicon does not seem to use the native platform and need to use the commented out targeting using the toolchain.

Using the selection as is, looks like it works on Windows and Linux, but they both x86_64 systems, vs the Apple machine which is native aarch64. I have to use the commented section instead to force docker image to use aarch64 image, else the build does not work.

[Emilgardis]

The image.toolchain config is an array, you can specify multiple platforms/toolchains

It works like that by design, as I've described earlier

[Emilgardis]

So, you would do

image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu", "linux/amd64=x86_64-unknown-linux-gnu"]

[KarstenB]

It's unclear to me if you've been able to solve your problem, the way to immediately solve your issue of not being able to specify --platform=linux/arm64 is the config I shared earlier, does that work for now?

With that config it's even possible to specify precisely what platform to use

[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
# syntax = <platform>=<toolchain>
image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

When I try this, I get:

❯ cross --version
cross 0.2.5
Error: 
   0: failed to parse file `"bla/Cross.toml"` as TOML
   1: invalid type: map, expected a string for key `target.aarch64-unknown-linux-gnu.image` at line 2 column 14

[Emilgardis]

@KarstenB I should clarify, above only works on the main branch, image.toolchain has not been released yet

so, you'll need to install cross as cargo install cross --git https://github.com/cross-rs/cross