Change user org in Grafana #4

matrixik · 2018-01-04T13:02:29Z

haruki-yamanashi · 2018-01-05T04:18:27Z

Please add GRAFANA_USERS in docker-compose-metric.yml (and add variable for grafana_admin_project in .env, dotenv2 if necessary).

haruki-yamanashi · 2018-01-05T05:30:59Z

I could not create default database and dashboard for user who is configured by GRAFANA_USERS.

My configuration
docker-compose-metric.yml

  grafana-init:
    hostname: docker-host
    image: fest/grafana-init:${MON_GRAFANA_INIT_VERSION}
    environment:
      GRAFANA_USERNAME: ${MON_GRAFANA_USERNAME}
      GRAFANA_PASSWORD: ${MON_GRAFANA_PASSWORD}
      GRAFANA_ADMIN_USERNAME: ${MON_KEYSTONE_ADMIN_USER}
      GRAFANA_ADMIN_PASSWORD: ${MON_KEYSTONE_ADMIN_PASSWORD}
      LOGSTASH_FIELDS: "service=grafana-init"
      GRAFANA_USERS: '[{"user": "${MON_KEYSTONE_ADMIN_USER}", "password": "${MON_KEYSTONE_ADMIN_PASSWORD}", "email": "", "project": "monasca", "domain": "Default"}]'
    depends_on:
      - grafana

Log

# docker-compose -f docker-compose-metric.yml -f docker-compose-log.yml logs grafana-init
Attaching to monascadocker_grafana-init_1
grafana-init_1           | INFO:root:Opening a Grafana session...
grafana-init_1           | INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana
grafana-init_1           | INFO:root:Attempting to add configured datasource...
grafana-init_1           | INFO:root:Creating dashboard from file: /dashboards.d/07-nodes.json
grafana-init_1           | INFO:root:Creating dashboard from file: /dashboards.d/08-openstack.json
grafana-init_1           | INFO:root:Ending u'admin' session...
grafana-init_1           | INFO:root:Finished successfully.

As far as I checked the log, 'if block' in here (https://github.com/matrixik/monasca-docker/blob/fed1abee6e5be296f260970aad18864ede33a7e6/grafana-init/grafana.py#L181) is not executed because logging.info('Setting user "%s" organisation to "%s"', user['user'], org_name) isn't output and it seems that change_user_context is not executed though user['project'] and user['domain'] is configured.

matrixik · 2018-01-05T10:07:43Z

In GRAFANA_USERS: you are adding dashboards for admin, is that what you want?

You can get more logs from grafana-init container with adding LOG_LEVEL: "DEBUG" to grafana-init environment section.

matrixik · 2018-01-05T10:57:02Z

One more question: did you rebuild grafana-init localy on your computer? Maybe that's the reason it's not executing. I used your config and it's going into this if block.

matrixik · 2018-01-05T13:17:05Z

GRAFANA_USERS added in .env and dotenv2 files.
@haruki-yamanashi check if this way is fine for you.

matrixik · 2018-01-05T13:19:56Z

What you mean by grafana_admin_project? What should it contain or configure?

witekest · 2018-01-08T12:54:45Z

It must be fixed on master first.

haruki-yamanashi · 2018-01-09T05:39:38Z

In GRAFANA_USERS: you are adding dashboards for admin, is that what you want?

I also would like to add default datasource for GRAFANA_USERS. So source code which creates datasource (https://github.com/matrixik/monasca-docker/blob/73d35ab7efaf659b79c8ba3b36cb1c3cf8f0f469/grafana-init/grafana.py#L175-L179) should be located after if block (https://github.com/matrixik/monasca-docker/blob/73d35ab7efaf659b79c8ba3b36cb1c3cf8f0f469/grafana-init/grafana.py#L181) I think.

haruki-yamanashi · 2018-01-09T05:59:25Z

One more question: did you rebuild grafana-init localy on your computer? Maybe that's the reason it's not executing. I used your config and it's going into this if block.

Oh... I forgot to create image.
Now I built grafana-init container image and tested, but grafana-init container went down and following log was output.

# docker-compose -f docker-compose-metric.yml -f docker-compose-log.yml logs grafana-init
Attaching to monascadocker_grafana-init_1
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana
DEBUG:requests.packages.urllib3.connectionpool:"POST /login HTTP/1.1" 200 23
INFO:root:Opening a Grafana session...
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana
DEBUG:requests.packages.urllib3.connectionpool:"POST /login HTTP/1.1" 200 23
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/datasources HTTP/1.1" 200 2
DEBUG:root:existing datasources = []
INFO:root:Attempting to add configured datasource...
DEBUG:root:payload = {'access': 'proxy', 'name': 'monasca', 'jsonData': {'keystoneAuth': True}, 'url': 'http://monasca:8070/', 'type': 'monasca-datasource', 'isDefault': True}
DEBUG:requests.packages.urllib3.connectionpool:"POST /api/datasources HTTP/1.1" 200 37
DEBUG:root:Response: {u'message': u'Datasource added', u'id': 1}
INFO:root:Setting user "admin" organisation to "monasca@Default"
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/orgs/name/monasca%40Default HTTP/1.1" 403 31
DEBUG:__main__:Caught exception, retrying...
Traceback (most recent call last):
  File "/grafana.py", line 56, in f_retry
    return func(*args, **kwargs)
  File "/grafana.py", line 110, in change_user_context
    org.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 403 Client Error: Forbidden for url: http://grafana:3000/api/orgs/name/monasca%40Default

variables in source code:

grafana_users: [{u'project': u'monasca', u'domain': u'Default', u'password': u'<my_admin_password>', u'user': u'admin', u'email': u''}]
org: <Response [403]>
GRAFANA_URL: http://grafana:3000
org_name: monasca%40Default

.env configuration:

MON_GRAFANA_ADMIN_USER=grafana-admin
MON_GRAFANA_ADMIN_PASSWORD=password
MON_GRAFANA_USERS=[{"user": "admin", "password": "<my_admin_password>", "email": "", "project": "monasca", "domain": "Default"}]

haruki-yamanashi · 2018-01-09T06:00:36Z

GRAFANA_USERS added in .env and dotenv2 files.

Looks good for me.

haruki-yamanashi · 2018-01-09T06:11:08Z

What you mean by grafana_admin_project? What should it contain or configure?

I assumed that GRAFANA_USERS is configured in docker-compose-metric.yml, grafana-init section in your previous change as below.

GRAFANA_USERS: '[{"user": "${MON_GRAFANA_USERNAME}", "password": "${MON_GRAFANA_PASSWORD}", "email": "", "project": "${MON_GRAFANA_PROJECT}", "domain": "Default"}]'

And assumed that MON_GRAFANA_PROJECT is added in .env

But project is configurable in .env, MON_GRAFANA_USERS in you current change so it's not necessary to add variable for project of grafana admin user.

matrixik · 2018-01-15T12:22:26Z

I also would like to add default datasource for GRAFANA_USERS. So source code which creates datasource (https://github.com/matrixik/monasca-docker/blob/73d35ab7efaf659b79c8ba3b36cb1c3cf8f0f469/grafana-init/grafana.py#L175-L179) should be located after if block (https://github.com/matrixik/monasca-docker/blob/73d35ab7efaf659b79c8ba3b36cb1c3cf8f0f469/grafana-init/grafana.py#L181) I think.

Done

matrixik · 2018-01-15T13:05:24Z

Are you sure that

MON_GRAFANA_ADMIN_USER=grafana-admin
MON_GRAFANA_ADMIN_PASSWORD=password

is admin user with all privileges?

When you login with this credentials in browser are you able to manage other users?

matrixik · 2018-01-16T07:57:40Z

@haruki-yamanashi I uploaded new change that will use proper Grafana admin user now.

I found out that when user configured in Grafana and Keystone have same names they will clash and this user is loosing admin rights.

haruki-yamanashi · 2018-01-16T10:18:48Z

docker-compose-metric.yml

@@ -217,8 +217,8 @@ services:
 hostname: docker-host
 image: fest/grafana-init:${MON_GRAFANA_INIT_VERSION}
 environment:
- GRAFANA_ADMIN_USERNAME: ${MON_KEYSTONE_ADMIN_USER}
- GRAFANA_ADMIN_PASSWORD: ${MON_KEYSTONE_ADMIN_PASSWORD}
+ GRAFANA_ADMIN_USERNAME: ${MON_GRAFANA_ADMIN_USER}


Is it necessary to login as grafana-admin in here? (https://github.com/matrixik/monasca-docker/blob/fix/grafana-org/grafana-init/grafana.py#L164)

Yes, to change user organisation Grafana need to use user with admin privileges, not Keystone admin user.

haruki-yamanashi · 2018-01-16T10:41:17Z

Are you sure that

MON_GRAFANA_ADMIN_USER=grafana-admin
MON_GRAFANA_ADMIN_PASSWORD=password

is admin user with all privileges?
When you login with this credentials in browser are you able to manage other users?

Yes, Grafana admin user, grafana-admin (Main Org.) can manage Keystone admin user, e.g. admin user who belongs to admin@Default and monasca@Default by browser (Admin -> Global Users -> Edit)

haruki-yamanashi · 2018-01-16T10:55:32Z

I checked default dashboard and datasource is created for MON_GRAFANA_USERS correctly.
I review the code again in tomorrow.

haruki-yamanashi · 2018-01-17T01:43:22Z

grafana-init/grafana.py

-GRAFANA_USERS = [{'user': GRAFANA_USERNAME, 'password': GRAFANA_PASSWORD, 'email': ''}]
+GRAFANA_ADMIN_USERNAME = os.environ.get('GRAFANA_ADMIN_USERNAME', 'admin')
+GRAFANA_ADMIN_PASSWORD = os.environ.get('GRAFANA_ADMIN_PASSWORD', 'password')
+GRAFANA_USERS = [{


If GRAFANA_USERS is not set as environment variable, this is used for creating default dashboard and datasource for one user (on the other hand, if GRAFANA_USERS is set as environment variable, this variable can be used for multiple users). Now we can create default dashboard and datasource for user belongs to only mini-mon project by change GRAFANA_USERNAME and GRAFANA_PASSWORD. So I think it's better that project and domain are configurable by environment variable.
Means my idea is change as below.

in grafana.py

GRAFANA_PROJECT = os.environ.get('GRAFANA_PROJECT', 'mini-mon') GRAFANA_DOMAIN = os.environ.get('GRAFANA_DOMAIN', 'Default') GRAFANA_USERS = [{ 'user': GRAFANA_USERNAME, 'password': GRAFANA_PASSWORD, 'email': '', 'project': 'GRAFANA_PROJECT', 'domain': 'GRAFANA_DOMAIN', }]

in Dokerfile

GRAFANA_PROJECT=mini-mon \ GRAFANA_DOMAIN=Default

I'm not sure how to name this two because they are actually not Grafana project and domain but Keystone project and domain. And we create Grafana org name in https://github.com/Brandstetter/monasca-docker/pull/4/files#diff-b01ee5885d0570af470948405281ad37R177

Oh. exactly it's difficult to name these...
How about GRAFANA_KEYSTONE_PROJECT or like that?

Or, how about remove GRAFANA_ADMIN_USERNAME and GRAFANA_ADMIN_PASSWORD because now we can configure user by using GRAFANA_USERS as environment variable for single or multiple user.

Unfortunately GRAFANA_ADMIN_USERNAME and GRAFANA_ADMIN_PASSWORD are needed because we need to know exactly what Grafana user have admin privileges, without it we are unable to change organization with API.

Sorry, above my comment is incorrect.

Or, how about remove GRAFANA_USERNAME and GRAFANA_PASSWORD because now we can configure user by using GRAFANA_USERS as environment variable for single or multiple user.
And change GRAFANA_USERS as below.

GRAFANA_USERS = [{ 'user': mini-mon, 'password': password, 'email': '', 'project': 'mini-mon', 'domain': 'Default', }]

This is configured by environment variable.

Done, removed GRAFANA_USERNAME and GRAFANA_PASSWORD

haruki-yamanashi · 2018-01-17T01:46:59Z

grafana-init/README.md

+Default data source and dashboard are created for multiple
+users if this variable is set as proper JSON format. e.g.
+`'[{"user": "mini-mon", "password": "password", "email": "", "project": "mini-mon", "domain": "Default"}, {"user": "other-user", "password": "password", "email": "", "project": "monasca", "domain": "Default"}]'`.
+Default value is overwritten by `GRAFANA_USERNAME` and `GRAFANA_PASSWORD`.


When we would like to create for multiple users, can't be overwritten by GRAFANA_USERNAME and GRAFANA_PASSWORD.

Done. I changed this description.

haruki-yamanashi · 2018-01-17T02:02:59Z

grafana-init/grafana.py

@@ -127,6 +160,9 @@ def create_dashboard_payload(json_path):


 def main():
+ admin_session = Session()
+ login(admin_session, create_admin_login_payload())
+
 for user in create_login_payload():


I can't create default dashboard and datasource properly in following status. (means created for only admin user belongs to monasca project).

in .env

MON_GRAFANA_USERS=[{"user": "admin", "password": "<admin_password>", "email": "", "project": "monasca", "domain": "Default"}, {"user": "admin", "password": "<admin_password>", "email": "", "project": "admin", "domain": "Default"}]

As far as I checked the log, this issue is caused by this method (https://github.com/matrixik/monasca-docker/blob/fix/grafana-org/grafana-init/grafana.py#L171)

logs

# docker-compose logs grafana-init Attaching to monascadocker_grafana-init_1 grafana-init_1 | INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /login HTTP/1.1" 200 23 grafana-init_1 | INFO:root:Opening a Grafana session... grafana-init_1 | INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /login HTTP/1.1" 200 23 grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"GET /api/datasources HTTP/1.1" 200 2 grafana-init_1 | DEBUG:root:existing datasources = [] grafana-init_1 | INFO:root:Setting user "admin" organisation to "monasca@Default" grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"GET /api/orgs/name/monasca%40Default HTTP/1.1" 200 120 grafana-init_1 | DEBUG:root:Organisation "monasca@Default" id = 3 grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /api/user/using/3 HTTP/1.1" 200 41 grafana-init_1 | INFO:root:Attempting to add configured datasource... grafana-init_1 | DEBUG:root:payload = {'access': 'proxy', 'name': 'monasca', 'jsonData': {'keystoneAuth': True}, 'url': 'http://monasca:8070/', 'type': 'monasca-datasource', 'isDefault': True} grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /api/datasources HTTP/1.1" 200 37 grafana-init_1 | DEBUG:root:Response: {u'message': u'Datasource added', u'id': 1} grafana-init_1 | INFO:root:Creating dashboard from file: /dashboards.d/07-nodes.json grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /api/dashboards/db HTTP/1.1" 200 56 grafana-init_1 | DEBUG:root:Response: {u'status': u'success', u'version': 0, u'slug': u'node-dashboard'} grafana-init_1 | INFO:root:Creating dashboard from file: /dashboards.d/08-openstack.json grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /api/dashboards/db HTTP/1.1" 200 61 grafana-init_1 | DEBUG:root:Response: {u'status': u'success', u'version': 0, u'slug': u'openstack-dashboard'} grafana-init_1 | INFO:root:Ending u'admin' session... grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"GET /logout HTTP/1.1" 302 29 grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"GET /login HTTP/1.1" 200 None grafana-init_1 | INFO:root:Opening a Grafana session... grafana-init_1 | INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): grafana grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"POST /login HTTP/1.1" 200 23 grafana-init_1 | DEBUG:requests.packages.urllib3.connectionpool:"GET /api/datasources HTTP/1.1" 200 461 grafana-init_1 | DEBUG:root:existing datasources = [{u'encryptedFields': None, u'name': u'monasca', u'database': u'', u'url': u'http://monasca:8070/', u'basicAuth': False, u'jsonData': {u'keystoneAuth': True}, u'access': u'proxy', u'isDefault': True, u'typeLogoUrl': u'public/plugins/monasca-datasource/img/openstack_logo.png', u'orgId': 3, u'user': u'', u'tlsAuth': {u'tlsClientKeySet': False, u'tlsClientCertSet': False, u'tlsCACertSet': False}, u'basicAuthUser': u'', u'password': u'', u'type': u'monasca-datasource', u'id': 1, u'basicAuthPassword': u'', u'withCredentials': False}] grafana-init_1 | INFO:root:Grafana has already been initialized, skipping!

I think this is solved by change line (location) of check_initialized method.

Oh, so you want to upload dashboards to two organizations of one user, I need to think a little more how to handle this properly.

At this moment, to upload dashboards to two organizations of one user is not necessary.
So it's OK for me that let this upstream.

haruki-yamanashi · 2018-01-17T02:06:54Z

grafana-init/grafana.py

@@ -127,6 +160,9 @@ def create_dashboard_payload(json_path):


 def main():
+ admin_session = Session()
+ login(admin_session, create_admin_login_payload())


I think it's necessary to end session also for GRANANA_ADMIN (e.g. grafana-admin) user.

witekest · 2018-01-18T13:24:56Z

grafana-init/grafana.py

-GRAFANA_USERS = [{'user': GRAFANA_USERNAME, 'password': GRAFANA_PASSWORD, 'email': ''}]
+GRAFANA_ADMIN_USERNAME = os.environ.get('GRAFANA_ADMIN_USERNAME', 'admin')
+GRAFANA_ADMIN_PASSWORD = os.environ.get('GRAFANA_ADMIN_PASSWORD', 'password')
+GRAFANA_USERS = [{


This should be set from environment variable.

It is set from env variable, check def create_login_payload(): in lines 70:80

Correct, thanks, haven't seen that.

haruki-yamanashi · 2018-01-19T07:54:16Z

@matrixik
Seems good from me, +1.
Thanks for your continuous updates.
Could you investigate/fix for Travis?
Then please let Brandstetter know and ask to merge.

@Brandstetter
Please final check and merge when Travis issue is fixed.

When Grafana and Keystone have configured user with same name they will clash and user in Grafana is loosing admin rights.

Update `grafana-init` README details about default grafana user env variables.

Signed-off-by: Dobroslaw Zybort <[email protected]>

matrixik · 2018-01-19T08:36:40Z

Looking into Travis problem now.

matrixik · 2018-01-19T12:44:23Z

Travis in PR will most likely fail in this fork with timeout when running tempest test, code is merged now into stable/pike.

haruki-yamanashi reviewed Jan 16, 2018

View reviewed changes

haruki-yamanashi reviewed Jan 17, 2018

View reviewed changes

witekest suggested changes Jan 18, 2018

View reviewed changes

matrixik added 7 commits January 19, 2018 09:32

Change user org in Grafana

24c98df

Expose full configuration of Grafana users in dotenv

e35f880

Update Grafana-init docs with proper users format

964c36d

Move adding Grafana datasource after changing user org

f8db60c

Change default Grafana admin user

cee0205

When Grafana and Keystone have configured user with same name they will clash and user in Grafana is loosing admin rights.

Logout Grafana admin user

ba2ad79

Update `grafana-init` README details about default grafana user env variables.

Bump grafana-init version

98eef2f

matrixik added 2 commits January 19, 2018 09:32

Bump version of Grafana-init in env files

c7def85

Remove GRAFANA_USERNAME and GRAFANA_PASSWORD from grafana-init

3497559

Signed-off-by: Dobroslaw Zybort <[email protected]>

matrixik force-pushed the fix/grafana-org branch from 78803ac to 3497559 Compare January 19, 2018 08:33

matrixik merged commit 10f64e6 into FujitsuEnablingSoftwareTechnologyGmbH:stable/pike Jan 19, 2018

Change user org in Grafana #4

Change user org in Grafana #4

Conversation

matrixik commented Jan 4, 2018

haruki-yamanashi commented Jan 5, 2018

haruki-yamanashi commented Jan 5, 2018 • edited Loading

matrixik commented Jan 5, 2018

matrixik commented Jan 5, 2018

matrixik commented Jan 5, 2018

matrixik commented Jan 5, 2018

witekest commented Jan 8, 2018

haruki-yamanashi commented Jan 9, 2018 • edited Loading

haruki-yamanashi commented Jan 9, 2018

haruki-yamanashi commented Jan 9, 2018

haruki-yamanashi commented Jan 9, 2018

matrixik commented Jan 15, 2018 • edited Loading

matrixik commented Jan 15, 2018

matrixik commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haruki-yamanashi commented Jan 16, 2018

haruki-yamanashi commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haruki-yamanashi Jan 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haruki-yamanashi commented Jan 19, 2018

matrixik commented Jan 19, 2018

matrixik commented Jan 19, 2018

haruki-yamanashi commented Jan 5, 2018 •

edited

Loading

haruki-yamanashi commented Jan 9, 2018 •

edited

Loading

matrixik commented Jan 15, 2018 •

edited

Loading

haruki-yamanashi Jan 18, 2018 •

edited

Loading