Emit an event when the result of a probe for a container changes

[intUnderflow]

/kind feature
/kind api-change
/sig node
/hold

What this PR does / why we need it:

This PR introduces a new event that is emitted by the probe worker when the result of the probe changes. End-users can watch this new event to learn when their probes entered the Sucess or Failure state and in the case of Failure they gain some contextual information in the form of the output from the last probe.

Fixes #115823 by dealing with the lack of information on when a probe fails (as users can now just look at this event)

Does this PR introduce a user-facing change?

A new event is emitted on containers when their readiness, liveness or startup probes change result from Success to Failure or vice-versa. In the case of Failure information on why the probe failed may be included.

[k8s-ci-robot]

Hi @intUnderflow. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[intUnderflow]

/cc @SergeyKanzhelev @andrewsykim @krmayankk

[RobertKielty]

/ok-to-test

[intUnderflow]

Had some thoughts:

• If this does require some work on the API serialisation side its probably best to break that up into another PR anyway
• This is still delivering something useful, the events are now annotated with belowFailureThreshold

Given that I think this is ready to go
/unhold

[dgrisonnet]

/cc

[k8s-triage-robot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[intUnderflow]

/retest

[dgrisonnet]

As far as I can tell this will not work as you expect. I believe that what you expect to see is the Event going from below threshold to above threshold after some failures right?

If that is the case, this is not what you will see on the apiserver side because of how we aggregate Events in the client. When merging similar Events together the client disregards the annotations and it is completely intended because annotations are not part of what we considered to be the structure of an Event. What will happen in practice is that the client will merge the existing emissions of below threshold Events for your Pod with the newer above threshold Event and emit a below threshold Event instead of an above threshold one.

That said, have you perhaps considered emitting a new Event instead of reusing an existing one?

[intUnderflow]

That said, have you perhaps considered emitting a new Event instead of reusing an existing one?

I have very little context in this area, but if you have any ideas as to how we consider the design around new events / etc I can start to put together a KEP and refactor this PR around that?

[SergeyKanzhelev]

when I looked at it I imagined two events - one for failed probe under the threshold and another - after. Those should be different events so the first will not make the second to be throttled.

As for tracking specific failure trends - it is indeed the best done via metrics.

[intUnderflow]

gentle ping @mrunalp @derekwaynecarr @dchen1107 @klueska

[intUnderflow]

@SergeyKanzhelev propose to move to priority/important-longterm, are you happy with this given you initially triaged the underlying issue?

Basing off https://www.kubernetes.dev/docs/guide/issue-triage/

[intUnderflow]

/priority important-longterm

[intUnderflow]

/remove-priority backlog

[intUnderflow]

(merge conflicts)

[SergeyKanzhelev]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: a29a3294e7e615092f744c3d79c4faaef410f1cc

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: intUnderflow, SergeyKanzhelev
Once this PR has been reviewed and has the lgtm label, please ask for approval from dchen1107. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[intUnderflow]

/retest

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Reopen this PR with /reopen
• Mark this PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Reopen this PR with /reopen
• Mark this PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.