feat: add CustomResourceMonitor CRD (WIP) #2049

CatherineF-dev · 2023-04-17T21:21:07Z

What this PR does / why we need it:
What: A more user-friendly UX to collect custom resource metrics.
Why:

Using CustomResourceState monitoring feature is not user-friendly. Current ways on configuring CustomResourceState monitoring are:

--custom-resource-state-config "inline yaml (see example)" or
--custom-resource-state-config-file /path/to/config.yaml

Current CustomResourceState monitoring feature doesn't support multiple configuration files.
For example, for a company with 10 teams, each team wants to collect CustomResourceState metrics for their owned Custom Resources.

See more in design doc.

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality) no

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2050 #1948

Design doc is ready to review while codes are proof-of-concepts.
Tested: 1. apply examples/cr.yaml 2. apply examples/cr2.yaml 3. delete two yamls.

CatherineF-dev · 2023-04-17T21:22:49Z

fyi: @chrischdi

rexagod · 2023-04-18T07:17:43Z

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

chrischdi · 2023-04-18T07:54:36Z

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

Let's move that discussion over to the issue itself #1948, happy to collect the facts and opinions there to properly document a decision :-)

Thanks @CatherineF-dev , I'll try to figure out time next week after kubecon to take a look at it 👍 🥳

CatherineF-dev · 2023-04-18T12:10:59Z

Hey, thank you for working on this. Just a question right off the top of my mind, would this be better suited to be in a KSM operator? I remember a discussion started my @mrueg in the room highlighting the need for this feature a while back but we eventually held off on adding it for the sole reason of setting up an operator first, IIRC.

The reason why for this PR is that coupled custom resources metrics targets and kube-state-metrics deployment causes managing custom resource configuration not easy. And then I found a CRD can handle this issue and also solve Read custom resource metrics configuration from a Custom Resource Definition #1948.

Currently, one of these flags is added into kube-state-metrics deployment.

--custom-resource-state-config "inline yaml (see example)" or
--custom-resource-state-config-file /path/to/config.yaml

For example, for a company with two teams (monitoring platform team and application team), application team needs to change kube-state-metrics deployment managed by monitoring platform team.

This isn't make KSM as KSM operator fully. It won't add CRD for non custom resource metrics. I agree that CRD makes control KSM memory harder, since new metrics might be added overtime. A flag --custom_resources_ksm_cr_watched_labels is added to handle this case.
KSM already watches pods/endpoints and reconciles around autosharding. This PR only adds one additional watch on CustomResourceMonitor CR and update custom metrics store dynamically.

CatherineF-dev · 2023-04-18T12:20:59Z

docs/design/custom-resource-metrics-crd.md

+
+```yaml
+# example cr
+apiVersion: customresource.ksm.io/v1alpha1


I find *.k8s.io needs additional approvals. Planned to use customresource.ksm.k8s.io before.

I'd prefer *.k8s.io group for the CR given we exist under the same organization here.

CatherineF-dev · 2023-06-13T15:24:16Z

Hi @dgrisonnet, could you help review this design doc when you're available? Thanks!

Related issues #2050 #1948

dgrisonnet · 2023-06-13T15:33:55Z

Sure, I'll try to catch up on the various discussion asap

logicalhan

/assign

dgrisonnet · 2023-06-27T16:57:12Z

/triage accepted
/assign

mrueg · 2023-07-03T08:16:43Z

docs/design/custom-resource-metrics-crd.md

+Apart from existing two flags (`--custom-resource-state-config ` and `--custom-resource-state-config-file`), these three flags will be added: 
+* `--custom_resource_monitor_enabled`: whether watch CustomResourceMonitor CRs or not.
+* `--custom_resource_monitor_labels`: only watch CustomResourceMonitor with label=x. It's used to avoid double custom metrics collection when multiple kube-state-metrics are installed.
+* `--custom_resource_monitor_namespaces`: only watch CustomResourceMonitor under namespaces=x.


Should we consider a denylist for namespaces as well?
In case someone has a "platform KSM" (only selected namespaces) and a "product KSM" (everything else)?

guettli · 2023-09-04T10:14:10Z

@CatherineF-dev I think it would be createa to have a CustomResourceMonitor CRD. Are you still working on this PR?

If you think this PR is not needed any more, because you found an alternative solution, please let me know. I guess we have the same goal.

k8s-triage-robot · 2024-01-21T10:19:53Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

rexagod · 2024-01-21T18:50:43Z

@CatherineF-dev Are you working on this? I see how this could be a large-enough issue entailing multiple cases to warrant more hands, and if so, feel free to bring this up in the next SIG call and we can have more folks contributing to your fork, if that sounds good!

CatherineF-dev · 2024-01-22T00:01:21Z

Yes, from previous discussion, I remember we plan to split KSM and KSM Custom Metrics into 2 binaries first. Then add CRD feature.

So I haven't resolved conflicts in this PR. Waiting 2 binaries.

(This prototype was working in my k8s cluster)

k8s-triage-robot · 2024-02-21T00:41:31Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

rexagod · 2024-02-25T19:26:22Z

I believe we can get this in regardless, and include this change whenever the split happens. Getting this one in would resolve a lot of issues, in addition to deserving its own minor release.

I'm +1 for making this happen pre-split, since we don't plan on doing that in the immediate future.

rexagod · 2024-02-25T19:26:33Z

/remove-lifecycle rotten

mv discovery to app use interface to decouple customresourcestate and CRDiscoverer

Co-authored-by: Manuel Rüger <[email protected]>

k8s-ci-robot · 2024-06-04T00:40:57Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 17, 2023

k8s-ci-robot requested review from dgrisonnet and logicalhan April 17, 2023 21:21

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 17, 2023

CatherineF-dev changed the title ~~Add Custom Resource Metrics CRD~~ Add CustomResourceMonitor CRD Apr 17, 2023

CatherineF-dev changed the title ~~Add CustomResourceMonitor CRD~~ [WIP] Add CustomResourceMonitor CRD Apr 17, 2023

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 17, 2023

CatherineF-dev force-pushed the custom-resource-crd branch 4 times, most recently from 0169068 to 88613ae Compare April 17, 2023 21:53

CatherineF-dev commented Apr 18, 2023

View reviewed changes

CatherineF-dev mentioned this pull request Apr 18, 2023

Decouple custom resource config with kube-state-metrics deployment #2050

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 20, 2023

bavarianbidi mentioned this pull request May 30, 2023

CR monitoring via KSM giantswarm/roadmap#2513

Open

logicalhan reviewed Jun 15, 2023

View reviewed changes

k8s-ci-robot assigned logicalhan Jun 15, 2023

CatherineF-dev force-pushed the custom-resource-crd branch 2 times, most recently from ad5da8a to b41f7d0 Compare June 25, 2023 14:31

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 25, 2023

CatherineF-dev changed the title ~~[WIP] Add CustomResourceMonitor CRD~~ feat: add CustomResourceMonitor CRD (WIP) Jun 25, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 25, 2023

mrueg reviewed Jul 3, 2023

View reviewed changes

CatherineF-dev force-pushed the custom-resource-crd branch 2 times, most recently from e30e223 to c2f9fee Compare July 9, 2023 19:49

chrischdi mentioned this pull request Aug 28, 2023

Dynamically load configmaps for custom metrics configuration #2169

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 28, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 21, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 21, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 25, 2024

CatherineF-dev and others added 10 commits May 15, 2024 06:58

mv discovery and util to customresourcestate/customresource

28cd416

mv discovery to app use interface to decouple customresourcestate and CRDiscoverer

Add CustomResourceMonitor type and CRD

873f950

Add generated codes

8b7a818

Fix import

68d5e13

Update custom metrics store to support delete or add custom resource

a3643bd

Add example

79db746

Add design doc

bbda990

Update custom-resource-metrics-crd.md

4e13343

Update docs/design/custom-resource-metrics-crd.md

ad268dc

Co-authored-by: Manuel Rüger <[email protected]>

Fix presubmit

942e800

CatherineF-dev force-pushed the custom-resource-crd branch from c2f9fee to 942e800 Compare May 15, 2024 11:01

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 15, 2024

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add CustomResourceMonitor CRD (WIP) #2049

feat: add CustomResourceMonitor CRD (WIP) #2049

CatherineF-dev commented Apr 17, 2023 •

edited

CatherineF-dev commented Apr 17, 2023

rexagod commented Apr 18, 2023 •

edited

chrischdi commented Apr 18, 2023

CatherineF-dev commented Apr 18, 2023 •

edited

CatherineF-dev Apr 18, 2023 •

edited

rexagod Jun 13, 2023

CatherineF-dev commented Jun 13, 2023

dgrisonnet commented Jun 13, 2023

logicalhan left a comment

dgrisonnet commented Jun 27, 2023

mrueg Jul 3, 2023

guettli commented Sep 4, 2023 •

edited

k8s-triage-robot commented Jan 21, 2024

rexagod commented Jan 21, 2024

CatherineF-dev commented Jan 22, 2024 •

edited

k8s-triage-robot commented Feb 21, 2024

rexagod commented Feb 25, 2024 •

edited

rexagod commented Feb 25, 2024

k8s-ci-robot commented Jun 4, 2024

feat: add CustomResourceMonitor CRD (WIP) #2049

Are you sure you want to change the base?

feat: add CustomResourceMonitor CRD (WIP) #2049

Conversation

CatherineF-dev commented Apr 17, 2023 • edited

CatherineF-dev commented Apr 17, 2023

rexagod commented Apr 18, 2023 • edited

chrischdi commented Apr 18, 2023

CatherineF-dev commented Apr 18, 2023 • edited

CatherineF-dev Apr 18, 2023 • edited

Choose a reason for hiding this comment

rexagod Jun 13, 2023

Choose a reason for hiding this comment

CatherineF-dev commented Jun 13, 2023

dgrisonnet commented Jun 13, 2023

logicalhan left a comment

Choose a reason for hiding this comment

dgrisonnet commented Jun 27, 2023

mrueg Jul 3, 2023

Choose a reason for hiding this comment

guettli commented Sep 4, 2023 • edited

k8s-triage-robot commented Jan 21, 2024

rexagod commented Jan 21, 2024

CatherineF-dev commented Jan 22, 2024 • edited

k8s-triage-robot commented Feb 21, 2024

rexagod commented Feb 25, 2024 • edited

rexagod commented Feb 25, 2024

k8s-ci-robot commented Jun 4, 2024

CatherineF-dev commented Apr 17, 2023 •

edited

rexagod commented Apr 18, 2023 •

edited

CatherineF-dev commented Apr 18, 2023 •

edited

CatherineF-dev Apr 18, 2023 •

edited

guettli commented Sep 4, 2023 •

edited

CatherineF-dev commented Jan 22, 2024 •

edited

rexagod commented Feb 25, 2024 •

edited