Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add containerd config options #11080

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

spnngl
Copy link

@spnngl spnngl commented Apr 12, 2024

What type of PR is this?
/kind feature

What this PR does / why we need it:

Get more containerd configuration options, notably SElinux and debug socket.

Which issue(s) this PR fixes:

Fixes #11081

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`containerd_debug_address`: configure containerd debug socket address
`containerd_debug_level`: sets the containerd debug log level. Supported levels are: "trace", "debug", "info", "warn", "error", "fatal", "panic"
`containerd_debug_format`: sets the containerd log format ("text" or "json")
`containerd_debug_uid`: sets containerd debug socket uid
`containerd_debug_gid`: sets containerd debug socket uid

`containerd_enable_selinux`: enable containerd SElinux enforcement
`containerd_disable_apparmor`: disable containerd apparmor enforcement
`containerd_tolerate_missing_hugetlb_controller`: tolerate containerd huge pages controller missing
`containerd_disable_hugetlb_controller`: disable containerd huge pages controller
`containerd_image_pull_progress_timeout`: sets containerd image pull timeout

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 12, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: spnngl
Once this PR has been reviewed and has the lgtm label, please assign floryut for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 12, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @spnngl. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 12, 2024
level = "{{ containerd_debug_level }}"
format = "{{ containerd_debug_format }}"

Copy link
Member

@yankay yankay Apr 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI @spnngl

When using containerd config default, it outputs:

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

So, the default config should not be changed :-)

If we want to customize the config, we can use the code:

{% if containerd_debug_address %}
  address = "{{ containerd_debug_address }}"
{% else %}

And the code in roles/container-engine/containerd/defaults/main.yml

# some comments...
# containerd_debug_address: "/var/run/containerd/debug.sock"

The same as the other variables.
Thanks

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it in last push

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, I really disagree with that. It's of little use to match the output of containerd config default, because it's not like we can use that directly in our jinja template, so we have to update mostly manually anyway if containerd defaults change.

I'd rather have explicit defaults and a more straightforward template, with as little branching as possible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put containerd_debug_level back to its previous value and the others will be used only if defined by the user

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant is that we should not have to use is defined everywhere, and keeping the same structure that the output of containerd config default buy us nothing, so we should just have something like:

{% if containerd_debug_enabled %}
[debug]
  setting_1 = {{ containerd_default_value_for_setting_1 }}
  ...
{% endif %}

@yankay
Copy link
Member

yankay commented Apr 14, 2024

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 14, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 15, 2024
@spnngl spnngl requested a review from yankay April 15, 2024 14:03
@spnngl spnngl force-pushed the chore/containerd/config branch 3 times, most recently from 72f332a to 66ca060 Compare April 17, 2024 13:01
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 17, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 17, 2024
@mzaian
Copy link
Contributor

mzaian commented May 6, 2024

@spnngl please add a proper release note.

@spnngl
Copy link
Author

spnngl commented May 7, 2024

I just added it to the release-note section of this PR

@mzaian
Copy link
Contributor

mzaian commented May 13, 2024

@VannTen @yankay Could you please do a final review here.

@@ -8,7 +8,19 @@ oom_score = {{ containerd_oom_score }}
max_send_message_size = {{ containerd_grpc_max_send_message_size }}

[debug]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm mistaken, the debug socket is not enabled if this section is not present.
IMO, this whole section should be guarded with something like containerd_enabled_debug_socket, defaulting to false.
Production setup should presumably not have a debug socket lying around.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is already present in current kubespray config to customize log level

See: https://github.com/kubernetes-sigs/kubespray/blob/master/roles/container-engine/containerd/templates/config.toml.j2#L10

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The debug log level is not the same thing that the normal log level, is it not ?

I know that section is already present. What I mean is that it should not be enabled by default, as kubespray by default target production use cases, and having a debug socket enabled in production is not optimal.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It set the log level for everything, not just debug ones.
Same for the log format inside it.

Do you still want to disable this section entirely with a containerd_enabled_debug bool ? And no if defined inside it ?

See: https://github.com/containerd/containerd/blob/83031836b2cf55637d7abf847b17134c51b38e53/cmd/containerd/command/main.go#L348

It takes the config.Debug.Level if no cli params is used, cli flags have higher priorities but we do not use them in kubespray.

func setLogLevel(context *cli.Context, config *srvconfig.Config) error {
	l := context.GlobalString("log-level")
	if l == "" {
		l = config.Debug.Level
	}
	if l != "" {
		return log.SetLevel(l)
	}
	return nil
}

Config struct is found here and the debug section looks like this:

type Debug struct {
	Address string `toml:"address"`
	UID     int    `toml:"uid"`
	GID     int    `toml:"gid"`
	Level   string `toml:"level"`
	// Format represents the logging format. Supported values are 'text' and 'json'.
	Format string `toml:"format"`
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, this is so weird (or I totally misunderstood something). Does that mean there is no way to change the log level from the config without enabling the debug socket ?
I mean, I assumed that the "normal" log stream (which I presume goes to standard output) and the "debug" log stream (to the debug socket) where different, but are they in fact the same ?

I've tried to check containerd documentation but couldn't find explanation about this.


That said, I still thinks we should not enable the debug socket in production, (unless the debug socket is an unfortunate naming and it's in fact required ?). Since we handle the generation of the systemd unit for containerd we could inject the log level as flags... 🤔

That does seems a bit weird from containerd though, I feel like I'm missing something.

Comment on lines 49 to 55
# debug socket location: unix or tcp format
containerd_debug_address: ""
containerd_debug_level: ""
# logs format, supported values: text, json
containerd_debug_format: ""
containerd_debug_uid: 0
containerd_debug_gid: 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have some consistency here : either put the default for all settings, or for none.
And I'd lean towards documenting the defaults explicitely.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put it to the containerd config default as requested previously, I changed it back to what it was for containerd_debug_level and add the others only if they are defined by the user

level = "{{ containerd_debug_level }}"
format = "{{ containerd_debug_format }}"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, I really disagree with that. It's of little use to match the output of containerd config default, because it's not like we can use that directly in our jinja template, so we have to update mostly manually anyway if containerd defaults change.

I'd rather have explicit defaults and a more straightforward template, with as little branching as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Containerd: add debug and some CRI configuration options
5 participants