Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not showing recommandation with AWS managed prometheus #165

Open
cx-asafd opened this issue Nov 6, 2023 · 3 comments
Open

Not showing recommandation with AWS managed prometheus #165

cx-asafd opened this issue Nov 6, 2023 · 3 comments

Comments

@cx-asafd
Copy link

cx-asafd commented Nov 6, 2023

Describe the bug
Can see recommendation from eks manages prom

To Reproduce
Steps to reproduce the behavior:
krr simple -p "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/..../api/v1/query" --eks-managed-prom -n eu

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
Screenshot 2023-11-06 at 10 12 31

@cx-asafd
Copy link
Author

cx-asafd commented Nov 7, 2023

[ERROR] No Prometheus or metrics service found

@jakesky-centrica
Copy link

jakesky-centrica commented Nov 14, 2023

Facing a similar issue

│    24. │ prometh… │ prometh… │ 0    │ 0        │ DaemonS… │ node-e… │          │ unset   │ unset -> │         │ unset -> │ unset   │
│        │          │          │      │          │          │         │          │ -> ?    │ ?        │         │ ?        │ -> ?    │
│        │          │          │      │          │          │         │          │ (Prome… │ (Promet… │         │ (Promet… │ (Prome… │
│        │          │          │      │          │          │         │          │ not     │ not      │         │ not      │ not     │
│        │          │          │      │          │          │         │          │ found)  │ found)   │         │ found)   │ found) 
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api   
python3.10 krr.py simple --eks-managed-prom -p "https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx" -v --eks-managed-prom-region eu-west-2


 _____       _               _          _  _______  _____
|  __ \     | |             | |        | |/ /  __ \|  __ \
| |__) |___ | |__  _   _ ___| |_ __ _  | ' /| |__) | |__) |
|  _  // _ \| '_ \| | | / __| __/ _` | |  < |  _  /|  _  /
| | \ \ (_) | |_) | |_| \__ \ || (_| | | . \| | \ \| | \ \
|_|  \_\___/|_.__/ \__,_|___/\__\__,_| |_|\_\_|  \_\_|  \_\



Running Robusta's KRR (Kubernetes Resource Recommender) 1.6.0-dev
Using strategy: Simple
Using formatter: table

[11:08:53] WARNING  The timeframe duration provided is insufficient and will be overridden with 2.016. Kindly adjust     runner.py:236
                    --timeframe_duration to a value equal to or greater than 2.016.                                                   
           DEBUG    Found 10 clusters: arn:aws:eks:eu-west-1:927431155644:cluster/ex-complete,                         __init__.py:353                               
                    arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                                                  
           DEBUG    Current cluster: arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                  __init__.py:354
           DEBUG    Configured clusters: []                                                                            __init__.py:356
           INFO     Using clusters: ['arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx']                                 runner.py:190
           INFO     Listing scannable objects in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                       __init__.py:62
           DEBUG    Namespaces: *                                                                                       __init__.py:63
           DEBUG    Resources: *                                                                                        __init__.py:64
[11:08:54] DEBUG    Listing Deployments in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:154
           DEBUG    Listing Rollouts in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                               __init__.py:154
           DEBUG    Listing StatefulSets in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                           __init__.py:154
           DEBUG    Listing DaemonSets in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                             __init__.py:154
           DEBUG    Listing Jobs in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                   __init__.py:154
           DEBUG    Rollout API not available in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                      __init__.py:195
           DEBUG    Found 0 Job in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                    __init__.py:166
           DEBUG    Found 3 StatefulSet in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:166
           DEBUG    Found 22 Deployment in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                            __init__.py:166
           DEBUG    Found 6 DaemonSet in arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                              __init__.py:166
           INFO     Connecting to Prometheus for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx     prometheus_metrics_service.py:63
                    cluster                                                                                                           
           INFO     Using Prometheus at                                                               prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx for cluster arn:aws:eks:eu-west-2:124978694494:cluster/xxxxx                                     
           DEBUG    Prometheus not found: Couldn't connect to Prometheus found under                                      loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query)                                                                                                        
           INFO     Connecting to Victoria for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx       prometheus_metrics_service.py:63
                    cluster                                                                                                           
           INFO     Using Victoria at                                                                 prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-009aa349-df06-485c-8                                 
                    ed2-670fe67af9e6 for cluster arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                     
[11:08:55] DEBUG    Victoria Metrics not found: Couldn't connect to Victoria Metrics found under                          loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by PrometheusNotFound: Couldn't connect to Prometheus found under                                          
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query))                                                                                                       
           INFO     Connecting to Thanos for arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx cluster prometheus_metrics_service.py:63
           INFO     Using Thanos at                                                                   prometheus_metrics_service.py:79
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx for cluster arn:aws:eks:eu-west-2:xxxxx:cluster/xxxxx                                     
           DEBUG    Thanos not found: Couldn't connect to Thanos found under                                              loader.py:68
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/xxxxx                
                    Caused by PrometheusNotFound: Couldn't connect to Prometheus found under                                          
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx                 
                    Caused by HTTPError: 403 Client Error: Forbidden for url:                                                         
                    https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxx/api             
                    /v1/query))                                                                                                       
           ERROR    No Prometheus or metrics service found    
kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3-eks-4f4795d

Profile
Using the default profile

Expected behaviour - Concise error handling to highlight Prometheus is not connected due to permission issue.

Terminology - EKS managed Prometheus references should be termed Amazon Managed Prometheus.

@aantn
Copy link
Contributor

aantn commented Nov 16, 2023

@cx-asafd and @jakesky-centrica, thanks for reporting.

Is the issue definitely a permission one then, and not a discovery issue? If so, does adding the params here solve it?

Regarding the terminology, you're correct of course. @pavangudiwada can you please update the terminology from EKS managed Prometheus to Amazon Managed Prometheus?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants