Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correctly excluding a package? #472

Open
cidrbl0ck opened this issue Apr 13, 2023 · 25 comments · May be fixed by #664
Open

Correctly excluding a package? #472

cidrbl0ck opened this issue Apr 13, 2023 · 25 comments · May be fixed by #664

Comments

@cidrbl0ck
Copy link

cidrbl0ck commented Apr 13, 2023 •
edited

While this is not a bug, or I guess a real issue, I am trying to determine how to exclude a package.

The long story is: I'm needing to build a custom EE to include WinRM, PSRM, and the community.windows collection. Everything I read mentions to use the ee-supported-rhel8 as a BASE image. However building fails due to :

error: No package matches 'openshift-clients'
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1

Since I don't use Openshift I'd like to exclude it. Reading here: https://access.redhat.com/solutions/6999217. it says to format your context/ContainerFile as such:

ARG EE_BASE_IMAGE=registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8
ARG EE_BUILDER_IMAGE=registry.redhat.io/ansible-automation-platform-21/ansible-builder-rhel8

FROM $EE_BASE_IMAGE as galaxy
ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
USER root

ADD _build /build
WORKDIR /build


FROM $EE_BUILDER_IMAGE as builder
ADD _build/requirements.txt requirements.txt
RUN ansible-builder introspect --sanitize --user-pip=requirements.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
RUN assemble

FROM $EE_BASE_IMAGE
ARG PKGMGR_OPTS="--nodocs --setopt=install_weak_deps=0 --setopt=rhel-8-for-x86_64-appstream-rpms.excludepkgs=ansible-core"
USER root
COPY --from=builder /output/ /output/
RUN /output/install-from-bindep && rm -rf /output/wheels

However this doesn't work as I'm sure it's a syntax problem with Ansible-Builder versus Podman. I'm an AAP shop so all I know is ansible-builder.
I'm looking for information on how to exclude openshift-clients and any other un-needed package so I can build this image successfully.
@github-actions github-actions bot added the needs_triage New item that needs to be triaged label Apr 13, 2023
@Shrews
Copy link
Contributor

Shrews commented Apr 13, 2023

There is no built-in mechanism to builder to explicitly ignore a Python requirement.

I will note that the openshift-clients package doesn't seem to be a thing on PyPI: https://pypi.org/project/openshift-clients

However, the openshift-client package does exist: https://pypi.org/project/openshift-client

So looks like a spelling error in some Python requirements file from somewhere (possibly your user supplied requirements.txt from your EE, or even one of the dependencies you pull in?) If you run builder with increased verbosity (-v3 I think would do it), you should see the output of the introspect command from your Containerfile. It would list all of the coalesced Python requirements and which packages they come from. That might help you track down the misspelled package name.

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Apr 13, 2023 •
edited

EDIT Apologies, I typo'd openshift-clients to openshift-client above

Thanks Shrews, Building with the highest verbosity returns this:

+ /usr/bin/microdnf install -y --nodocs --setopt install_weak_deps=0 findutils gcc libcurl-devel libssh-devel libxml2-devel make openshift-clients openssl-devel python3-rpm python38-Cython python38-devel python38-lxml python38-pytz python38-pyyaml python38-requests python39-devel python39-lxml qemu-img
error: No package matches 'openshift-clients'
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1

Not super helpful but, somewhere theres a file telling microdnf to install this list of packages. Is that maybe 'embedded' in the BASE_IMAGE?

My requirements.txt doesn't exist in this case, and requirements.yml (even though prolly not related) contains 2 collections to be included.

There is an internal issue created back in mid-March with regards to the openshift-clients problem: https://access.redhat.com/solutions/6990615

Environment
Red Hat® Ansible Automation Platform 2.3
Issue
Why is ansible-builder unable to install the openshift-clients package even if it is present in the ee-supported Base Image in Ansible Automation Platform 2.3?
Resolution
This Issue is raised in our Internal Bug Tracking System and the Engineering Team is working on it.

Their workaround is to use the ee-minimal image, which I don't want to do.

@Shrews
Copy link
Contributor

Shrews commented Apr 13, 2023

Ah, this is a bindep dependency, not Python, as I had assumed.

Looking at the internal issues, I don't see any suggested, good resolution. But, IMO, this does seem to be a problem with one of the RH registry images.

@Akasurde Looks like you were helping look into this on the internal issues. Any information you could supply here?

@cidrbl0ck
Copy link
Author

I tried building again with a bindep.txt in the working dir, copy/pasted the bindep output from earlier and removed openshift-clients but the build failed.. my bindep is being ignored/ovwerwritten.

@tuxpreacher
Copy link

What does your execution-environment.yml look like? You specify the name/location of your files there.

@Akasurde
Copy link
Member

@cidrbl0ck Also, What is the content of requirements.yml?

@cidrbl0ck
Copy link
Author

So admittedly I did forget to put a reference to the bindep.txt file in execution-environments.yml. However correctign that and re-running the build resulted in the same thing.

---
version: 1

build_arg_defaults:
    EE_BASE_IMAGE: 'private_automation_hub/23-supported-rhel8:latest'

dependencies:
  galaxy: requirements.yml
  system: bindep.txt

Bindep.txt:

gcc
libcurl-devel
libssh-devel
libxml2-devel
make
openssl-devel
python3-rpm
python38-Cython
python38-devel
python38-lxml
python38-pytz
python38-pyyaml
python38-requests
python39-devel
python39-lxml
qemu-img

@cidrbl0ck
Copy link
Author

Requirements.yml:

collections:
   - community.windows
   - community.general

@cidrbl0ck
Copy link
Author

Checking the failed results further, there's another reference to openshift-clients:

  • 'openshift-clients [platform:rhel-8] # from collection kubernetes.core'

But that's additional to the bindep line.. I think. Maybe the actual problem is in that collection?

@cidrbl0ck
Copy link
Author

Update: So using buildah I modified the standard ansible-automation-platform23-ee-supported-rhel8 image to remove:
Collections:
kubernetes.core
arista*
ibm*

Python Packages:
openshift-clients

Then installed:
community.windows collection

Committing using buildah alleviated the task of building with ansible-builder, pushed it to my private automation hub and it's been used so far in roughly 40 deploys between 6 templates. All Windows, no issues.

Stinks that there's no methods of excluding in AB, also stinks that whatever was included in the BASE image wholly ignored by bindep.txt, and requirements.txt. This might not be the most elegant solution but it certainly worked until such time as the underlying openshift-clients issue is resolved in ee-supported-rhel8.

@sivel sivel removed the needs_triage New item that needs to be triaged label Jun 1, 2023
@madorn madorn mentioned this issue Jul 12, 2023
Merged
@fletort
Copy link

fletort commented Nov 17, 2023 •
edited

I have also this problem... Building an EE image from an ubi8 base image. I am adding the kubernetes.core collection so the assemble step failed as the rpm 'openshift-clients' is missing.
But i don"t need openshift feature of the kubernetes.core collection... only the kubernetes feature....; It is a mess because only one module of this collection need this dependency..; no the other one.....

We should have a way to block some dependencies listed by introspect...

I will try to find a way to install the collection after the intrrospect step, maybe in the final build stage...

@cidrbl0ck
Copy link
Author

Just an update for anyone fighting the same problem, since the issue here is 6 months old and there's no progress. (Admittedly this is a very small issue) Since I do not build EE's for distribution anywhere outside out internal Private Automation Hub I have continued to use Buildah with zero problems.
I realize there's some Ansiblthonic-ness that I'm willfully excluding but there have been no drawbacks from this method so it will continue into the foreseeable future.

@sivel sivel mentioned this issue Feb 6, 2024
Closed
@kbreit
Copy link

kbreit commented Feb 23, 2024

I'm experiencing a need for excluding bindep packages. In my case a collection is trying to install python which doesn't exist and I want to exclude it. Outside of the suggested Buildah approach, I haven't figured out a way to make this work.

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@kbreit-insight
Copy link

You can’t just remove the collection with the python requirement?

The collection is an important piece of the automation so I can't drop the collection.

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@kbreit-insight
Copy link

Just wanting to understand so I can help. If the collection requires Python.. but python isn’t available. Then I assume that does not break the entire collection but possibly parts that you don’t use?

Python is installed on the EE. Specifically, I'm installing https://github.com/ansible/galaxy_collection which, in bindep, appears to try to install the package named python. The AAP RHEL8 minimal image, which I'm using, includes python3.9 out of the box.

I started a container using podman and here's the output if I try to install python.

bash-4.4# microdnf install python
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
error: No package matches 'python'

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@kbreit-insight
Copy link

You didn’t say but, I assume you are installing it via Ansible-galaxy yeah?

Mostly correct. I'm installing it using ansible-galaxy but I'm actually installing it directly from GitHub because there's a bug in the collection that is fixed in git but hasn't been released to galaxy.ansible.com.

collections:
  - name: git+https://github.com/ansible/galaxy_collection
    type: git

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@kbreit-insight
Copy link

Gotcha. Im curious.. have you given the supported image a try.. just for curiosities sake? There’s around 50 packages in the supported that’s not in the minimal… if it was me and I really wanted to use the minimal I’d try the other just to see. But in any case.. other than perhaps the python syntax I mentioned earlier I don’t think I can help much more.

The supported EE is not as well suited for my use case as image size is important for my situation. However, I gave it a try anyways and it failed with the same error as the python package doesn't exist for most/all RHEL environments.

My feeling is there are 3 options to fix it in descending order of quality. The Buildah options is in here somewhere as well.

  1. Update the collection's bindep file to be more precise about the distributions that package should be installed for.
  2. Add support for excluding packages in ansible-builder.
  3. It seems the EE's have an /etc/dnf/dnf.conf file which should be able to take an excludes= option. However, I'm not seeing when it's created so more investigation is needed. This may be the fastest way to fix the problem though.

@cidrbl0ck
Copy link
Author

cidrbl0ck commented Feb 23, 2024 via email

@kbreit-insight
Copy link

And you are positive that the role/collection attempts to install python.. just that name exactly right? If so that sounds like a bug to me.. unless the collection was never meant for an RBI? Microdnf obviously doesn’t match on search/repoquery terms like full-fledged dnf does. I’m not a huge Python guy but does removing/commenting out the python line in the bindep.txt file help at all?

If I comment out the collection from requirements.yml the EE builds fine. I hope it is supposed to work with a UBI becaseu moving to the Red Hat supported automation_hub one will be tough unless they release a new version with the bug fix I need.

Regarding the bindep file, my next step on Monday is to create a fork, kill that file, and see if that works as an interm solution.

@sivel sivel linked a pull request Mar 26, 2024 that will close this issue
Open
5 tasks
@Shrews Shrews linked a pull request Apr 12, 2024 that will close this issue
Expose a way to exclude dependencies #664
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Expose a way to exclude dependencies Shrews/ansible-builder
8 participants
@sivel @fletort @Akasurde @Shrews @kbreit @tuxpreacher @cidrbl0ck @kbreit-insight