Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libbpf dependency of iproute2 is too old #103

Open
zhaiyan920 opened this issue Aug 17, 2023 · 3 comments
Open

libbpf dependency of iproute2 is too old #103

zhaiyan920 opened this issue Aug 17, 2023 · 3 comments

Comments

@zhaiyan920
Copy link

Hi,

I was encountering some relocation issue when test PR on BPF-CI. After some digging the error says:

libbpf: prog 'test_lwt_redirect_in': bad map relo against '.rodata.str1.1' in section '.rodata.str1.1'

It turns out the latest root image libbpf-vmtest-rootfs-2022.10.23-bullseye-amd64.tar.zst has iproute2 with libbpf-0.3, and support for .rodata. was introduced in commit 5bf62459b185 ("libbpf: Support multiple .rodata.* and .data.* BPF maps") from v0.6.0. Unpacking and running the root image locally with same BPF object confirms that upgrading libbpf0 would fix the issue.

Will it be feasible if admins can update the root image to pull libbpf0 from bullseye-backport? It does have slight newer version https://packages.debian.org/bullseye-backports/libbpf0, or better if we can have newer iproute2 and libbpf1 in the root image?

thanks
zhaiyan920 pushed a commit to zhaiyan920/bpf-ci that referenced this issue Aug 17, 2023
selftests/bpf: fix BPF-CI issue
eefa61e 
BPF-CI's iproute2 is too old:

libbpf/ci#103

Some workaround to avoid running into the relocation issue.

Signed-off-by: Yan Zhai <[email protected]>
zhaiyan920 pushed a commit to zhaiyan920/bpf-ci that referenced this issue Aug 17, 2023
selftests/bpf: fix BPF-CI issue
ee5396a 
BPF-CI's iproute2 is too old:

libbpf/ci#103

Some workaround to avoid running into the relocation issue.

Signed-off-by: Yan Zhai <[email protected]>
@chantra
Copy link
Collaborator

chantra commented Nov 4, 2023

Thanks @zhaiyan920 for reporting this, and sorry for getting to it so late.

In the future, we may not need to use the rootfs as part of the CI, so this should be resolved at the same time.

@jerryz920
Copy link

It's all good as I worked around the issue with some variant implementation. Just curious, what alternatives do you plan to replace rootfs? Is it possible we use 9p and virtio instead in future?

@chantra
Copy link
Collaborator

chantra commented Nov 5, 2023 via email

chantra added a commit to chantra/libbpf-ci that referenced this issue Jan 11, 2024
@chantra
run-vmtest: action to test bpf using danobi/vmtest
52dee0f 
Add a new action to mimick what used to be done by `run-qemu`.

This action is roughly a `cp run-qemu run-vmtest` in term of functionalities.
See end of this commit message for a rationale of this change.

Just like `run-qemu` assumes the presence of a rootfs which is provisioned with
a copy of the script from `ci/vmtest/run_selftests.sh`, `run-vmtest` assumes the
presence of `ci/vmtest/vmtest_selftests.sh` and will run it.

`vmtest_selftests.sh` functionally does the same as `run_selftests.sh` with a
few adjustments to make it work in the `vmtest` world` and leave in the "callee"
repository, not libbpf-ci.

`print_test_summary.py` was copied over unchanged. Later diff will remove the
version from `run-qemu` and point to this one instead.

`action.yml` needs to install a few tools that were historically baked in the rootfs.
A bunch of parameters are now historical.... this diff does not attempt to remove them
yet. This will be address later too and will probably change as libbpf/libbpf use case
is merged in.

`run.sh` gets rid of the logic to create the `qemu` one-liner as well as the
downloading of files from within the rootfs, and adjust it to using files on
disk that were left over by the test run.

Full end-to-end testing will be done through a PR in https://github.com/kernel-patches/vmtest

== Main functional difference between `run-qemu` and `run-vmtest`.

`run-qemu` runs the kernel inside a rootfs which is isolated from the host FS,
meaning that we potentially have a host in which we build the kernel/selftests
which is different than the host we run in. Causing depedency issues, see libbpf#83 and
to some extends libbpf#103 .
We could work around this with statically built binaries, but because of the different
rootfs, we end up having to maintain rootfs *and* doing a dance of copying files
in and out of the rootfs (through the prepare-rootfs action), which is slow, pulls
a fair amount of bytes through the network...

`run-vmtest` on the other hand does mount the host rootfs (read-only) and shares the
current directory (read-write) under /mnt/vmtest.
This resolves 2 things for us.
1. We don't have libraries incompatibilitie issues anymore as the building OS
and the OS running the tests will be the same.
1. We don't need to copy files in and out of the root fs. The files are already
on the FS seen by the VM, and files dumped by the tests are directly accessible
outside the VM.

On top of this, `vmtest` also handles the peculiarities of crafting the right
qemu one-liner.

Signed-off-by: Manu Bretelle <[email protected]>
@chantra chantra mentioned this issue Jan 12, 2024
Merged
chantra added a commit to chantra/libbpf-ci that referenced this issue Jan 25, 2024
@chantra
run-vmtest: action to test bpf using danobi/vmtest
91765e0 
Add a new action to mimick what used to be done by `run-qemu`.

This action is roughly a `cp run-qemu run-vmtest` in term of functionalities.
See end of this commit message for a rationale of this change.

Just like `run-qemu` assumes the presence of a rootfs which is provisioned with
a copy of the script from `ci/vmtest/run_selftests.sh`, `run-vmtest` assumes the
presence of `ci/vmtest/vmtest_selftests.sh` and will run it.

`vmtest_selftests.sh` functionally does the same as `run_selftests.sh` with a
few adjustments to make it work in the `vmtest` world` and leave in the "callee"
repository, not libbpf-ci.

`print_test_summary.py` was copied over unchanged. Later diff will remove the
version from `run-qemu` and point to this one instead.

`action.yml` needs to install a few tools that were historically baked in the rootfs.
A bunch of parameters are now historical.... this diff does not attempt to remove them
yet. This will be address later too and will probably change as libbpf/libbpf use case
is merged in.

`run.sh` gets rid of the logic to create the `qemu` one-liner as well as the
downloading of files from within the rootfs, and adjust it to using files on
disk that were left over by the test run.

Full end-to-end testing will be done through a PR in https://github.com/kernel-patches/vmtest

== Main functional difference between `run-qemu` and `run-vmtest`.

`run-qemu` runs the kernel inside a rootfs which is isolated from the host FS,
meaning that we potentially have a host in which we build the kernel/selftests
which is different than the host we run in. Causing depedency issues, see libbpf#83 and
to some extends libbpf#103 .
We could work around this with statically built binaries, but because of the different
rootfs, we end up having to maintain rootfs *and* doing a dance of copying files
in and out of the rootfs (through the prepare-rootfs action), which is slow, pulls
a fair amount of bytes through the network...

`run-vmtest` on the other hand does mount the host rootfs (read-only) and shares the
current directory (read-write) under /mnt/vmtest.
This resolves 2 things for us.
1. We don't have libraries incompatibilitie issues anymore as the building OS
and the OS running the tests will be the same.
1. We don't need to copy files in and out of the root fs. The files are already
on the FS seen by the VM, and files dumped by the tests are directly accessible
outside the VM.

On top of this, `vmtest` also handles the peculiarities of crafting the right
qemu one-liner.

Signed-off-by: Manu Bretelle <[email protected]>
chantra added a commit that referenced this issue Jan 25, 2024
@chantra
run-vmtest: action to test bpf using danobi/vmtest
740c566 
Add a new action to mimick what used to be done by `run-qemu`.

This action is roughly a `cp run-qemu run-vmtest` in term of functionalities.
See end of this commit message for a rationale of this change.

Just like `run-qemu` assumes the presence of a rootfs which is provisioned with
a copy of the script from `ci/vmtest/run_selftests.sh`, `run-vmtest` assumes the
presence of `ci/vmtest/vmtest_selftests.sh` and will run it.

`vmtest_selftests.sh` functionally does the same as `run_selftests.sh` with a
few adjustments to make it work in the `vmtest` world` and leave in the "callee"
repository, not libbpf-ci.

`print_test_summary.py` was copied over unchanged. Later diff will remove the
version from `run-qemu` and point to this one instead.

`action.yml` needs to install a few tools that were historically baked in the rootfs.
A bunch of parameters are now historical.... this diff does not attempt to remove them
yet. This will be address later too and will probably change as libbpf/libbpf use case
is merged in.

`run.sh` gets rid of the logic to create the `qemu` one-liner as well as the
downloading of files from within the rootfs, and adjust it to using files on
disk that were left over by the test run.

Full end-to-end testing will be done through a PR in https://github.com/kernel-patches/vmtest

== Main functional difference between `run-qemu` and `run-vmtest`.

`run-qemu` runs the kernel inside a rootfs which is isolated from the host FS,
meaning that we potentially have a host in which we build the kernel/selftests
which is different than the host we run in. Causing depedency issues, see #83 and
to some extends #103 .
We could work around this with statically built binaries, but because of the different
rootfs, we end up having to maintain rootfs *and* doing a dance of copying files
in and out of the rootfs (through the prepare-rootfs action), which is slow, pulls
a fair amount of bytes through the network...

`run-vmtest` on the other hand does mount the host rootfs (read-only) and shares the
current directory (read-write) under /mnt/vmtest.
This resolves 2 things for us.
1. We don't have libraries incompatibilitie issues anymore as the building OS
and the OS running the tests will be the same.
1. We don't need to copy files in and out of the root fs. The files are already
on the FS seen by the VM, and files dumped by the tests are directly accessible
outside the VM.

On top of this, `vmtest` also handles the peculiarities of crafting the right
qemu one-liner.

Signed-off-by: Manu Bretelle <[email protected]>
chantra added a commit to chantra/kernel-patches-vmtest that referenced this issue Jan 25, 2024
@chantra
[vmtest] using danobi/vmtest to run selftest
aa82f29 
https://github.com/danobi/vmtest make it easy to run a VM with a given kernel
and using the host rootfs (RO) and a shared directory (RW).

This will help us with a couple of issue:
- we don't have to maintain a rootfs anymore
- when running the test, we do not need to perform a dance with libguestfs, which allows us to save time during test, and make it trivial to exchange files between the host and the guest.
- we won't have anymore discrepency between the libraries used in the host and guest (like explained in libbpf/ci#84 and also libbpf/ci#103)

This change migrates the test runs from using `prepare-rootfs` + `run-qemu` actions to using `run-vmtest` instead.
@chantra chantra mentioned this issue Jan 25, 2024
Merged
chantra added a commit to kernel-patches/vmtest that referenced this issue Jan 26, 2024
@chantra
[vmtest] using danobi/vmtest to run selftest
1631e39 
https://github.com/danobi/vmtest make it easy to run a VM with a given kernel
and using the host rootfs (RO) and a shared directory (RW).

This will help us with a couple of issue:
- we don't have to maintain a rootfs anymore
- when running the test, we do not need to perform a dance with libguestfs, which allows us to save time during test, and make it trivial to exchange files between the host and the guest.
- we won't have anymore discrepency between the libraries used in the host and guest (like explained in libbpf/ci#84 and also libbpf/ci#103)

This change migrates the test runs from using `prepare-rootfs` + `run-qemu` actions to using `run-vmtest` instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@chantra @jerryz920 @zhaiyan920