monasca_persister can not acccess kafka

[zhangjianweibj]

hello,i run monasca-docker in one node.api and other components woks well.only monasca_persister not working. logs as follow：

i wonder all components run in one docker node.api and other components can access kafka,only persister can not access kafka.why?very thanks.

[matrixik]

Sorry, never had such problem. Look like it can't event connect to Kafka.
That's error from Python code so at the starting of container it should be able to connect properly to Kafka (connection is checked by start.sh script) and later something happen.
Do you have any errors/warnings printed by Kafka?
Did you change environment variable KAFKA_URI for persister?
Is this 172.19.0.3 proper IP?

[zhangjianweibj]

ok.very thanks.172.19.0.3 is a proper ip.other components wroks well.

[zhangjianweibj]

i just run "docker stack deploy -c docker-compose.yml",but monasca forward and other components can not accss each other.

i want to run monasca not in one node.so i want use docker swarm.but find "docker stack deploy -c docker-compose.yml" encountered many errors.can your give me some advises use docker-compose deploy monaca components on different nodes.very thanks

[matrixik]

As I wrote in other bug I never used Docker Swarm so can't help you with this problems...

[zhangjianweibj]

very thank.did your ever use monasca-docker tool, deploy the monasca components on different nodes?

[matrixik]

Nope, my part is to only make it working on one node. Sorry.

[matrixik]

I read a little about docker swarm and looks like it's capable of running from docker-compose.yml file. That's nice.
Did you make many changes to default docker compose files?
Could you share them?
Did you try running docker swarm with default docker-compose file (as is from this repository) to check if default configuration is working fine or also have this problems?

[zhangjianweibj]

version: '3'
services:

  memcached:
    image: memcached:1.5.0-alpine
    environment:
      LOGSTASH_FIELDS: "service=memcached"

  influxdb:
    image: influxdb:1.3.3-alpine
    environment:
      LOGSTASH_FIELDS: "service=influxdb"
  influxdb-init:
    image: monasca/influxdb-init:1.0.1
    environment:
      LOGSTASH_FIELDS: "service=influxdb-init"
    depends_on:
      - influxdb

  cadvisor:
    image: google/cadvisor:v0.27.1
    environment:
      LOGSTASH_FIELDS: "service=cadvisor"
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker:/var/lib/docker:ro"

  agent-forwarder:
    image: monasca/agent-forwarder:master-20180824-114532
    environment:
      NON_LOCAL_TRAFFIC: "true"
      LOGSTASH_FIELDS: "service=monasca-agent-forwarder"

  agent-collector:
    image: monasca/agent-collector:master-20180914-130916
    restart: on-failure
    environment:
      AGENT_HOSTNAME: "docker-host"
      FORWARDER_URL: "http://agent-forwarder:17123"
      CADVISOR: "true"
      CADVISOR_URL: "http://cadvisor:8080/"
      LOGSTASH_FIELDS: "service=monasca-agent-collector"
      MONASCA_MONITORING: "true"
      MONASCA_LOG_MONITORING: "false"
    cap_add:
      - FOWNER
    volumes:
      - "/:/rootfs:ro"

  alarms:
    image: monasca/alarms:1.2.0
    environment:
      LOGSTASH_FIELDS: "service=monasca-alarms"
    depends_on:
      - keystone
      - monasca

  zookeeper:
    image: zookeeper:3.4
    environment:
      LOGSTASH_FIELDS: "service=zookeeper"
    restart: on-failure

  kafka:
    image: monasca/kafka:0.9.0.1-2.11-1.1.6
    environment:
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      LOGSTASH_FIELDS: "service=kafka"
    restart: on-failure
    depends_on:
      - zookeeper
  kafka-watcher:
    image: monasca/kafka-watcher:0.0.3
    environment:
      BOOT_STRAP_SERVERS: "kafka"
      PROMETHEUS_ENDPOINT: "0.0.0.0:8080"
      LOGSTASH_FIELDS: "service=kafka-watcher"
    depends_on:
      - kafka
    ports:
      - "18080:8080"
  kafka-init:
    image: monasca/kafka-init:0.0.3
    environment:
      ZOOKEEPER_CONNECTION_STRING: "zookeeper:2181"
      KAFKA_TOPIC_CONFIG: segment.ms=900000 # 15m
      KAFKA_CREATE_TOPICS: "\
        metrics:64:1,\
        alarm-state-transitions:12:1,\
        alarm-notifications:12:1,\
        retry-notifications:3:1,\
        events:12:1,\
        kafka-health-check:1:1,\
        60-seconds-notifications:3:1"
      LOGSTASH_FIELDS: "service=kafka-init"
    depends_on:
      - zookeeper

  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: secretmysql
      LOGSTASH_FIELDS: "service=mysql"
  mysql-init:
    image: monasca/mysql-init:1.5.4
    environment:
      MYSQL_INIT_DISABLE_REMOTE_ROOT: "false"
      MYSQL_INIT_RANDOM_PASSWORD: "false"
      LOGSTASH_FIELDS: "service=mysql-init"

  keystone:
    image: monasca/keystone:1.1.3
    environment:
      KEYSTONE_HOST: keystone
      KEYSTONE_PASSWORD: secretadmin
      KEYSTONE_DATABASE_BACKEND: mysql
      KEYSTONE_MYSQL_HOST: mysql
      KEYSTONE_MYSQL_USER: keystone
      KEYSTONE_MYSQL_PASSWORD: keystone
      KEYSTONE_MYSQL_DATABASE: keystone
      LOGSTASH_FIELDS: "service=keystone"
    depends_on:
      - mysql
    ports:
      - "5000:5000"
      - "35357:35357"

  monasca-sidecar:
    image: timothyb89/monasca-sidecar:1.0.0
    environment:
      LOGSTASH_FIELDS: "service=monasca-sidecar"

  monasca:
    image: monasca/api:master-20180105-072513
    environment:
      SIDECAR_URL: http://monasca-sidecar:4888/v1/ingest
      LOGSTASH_FIELDS: "service=monasca-api"
    depends_on:
      - influxdb
      - keystone
      - mysql
      - zookeeper
      - kafka
      - monasca-sidecar
      - memcached
    ports:
      - "8070:8070"

  monasca-persister:
    image: monasca/persister:1.6
    environment:
      LOGSTASH_FIELDS: "service=monasca-persister"
    restart: on-failure
    depends_on:
      - monasca
      - influxdb
      - zookeeper
      - kafka

  storm-nimbus:
    image: monasca/storm:1.0.3
    command: storm nimbus
    environment:
      STORM_LOCAL_HOSTNAME: "storm-nimbus"
      WORKER_LOGS_TO_STDOUT: "true"
    depends_on:
      - zookeeper
      - kafka

  storm-supervisor:
    image: monasca/storm:1.0.3
    command: storm supervisor
    depends_on:
      - storm-nimbus
      - zookeeper
      - kafka

  thresh-init:
    image: monasca/thresh:master
    environment:
      STORM_WAIT_RETRIES: 50
    depends_on:
      - zookeeper
      - storm-nimbus
      - storm-supervisor

  monasca-notification:
    image: monasca/notification:master-20171024-085537
    environment:
      NF_PLUGINS: "webhook"
      LOGSTASH_FIELDS: "service=monasca-notification"
      STATSD_HOST: monasca-statsd
      STATSD_PORT: 8125
    depends_on:
      - monasca
      - zookeeper
      - kafka
      - mysql

  grafana:
    image: monasca/grafana:4.0.0-1.5.1
    environment:
      GF_AUTH_BASIC_ENABLED: "false"
      GF_USERS_ALLOW_SIGN_UP: "true"
      GF_USERS_ALLOW_ORG_CREATE: "true"
      GF_AUTH_KEYSTONE_ENABLED: "true"
      GF_AUTH_KEYSTONE_AUTH_URL: http://keystone:5000
      GF_AUTH_KEYSTONE_VERIFY_SSL_CERT: "false"
      GF_AUTH_KEYSTONE_DEFAULT_DOMAIN: "Default"
      LOGSTASH_FIELDS: "service=grafana"
    ports:
      - "3000:3000"
    depends_on:
      - keystone
      - monasca

  grafana-init:
    image: monasca/grafana-init:1.4.0
    environment:
      LOGSTASH_FIELDS: "service=grafana-init"
    depends_on:
      - grafana

  monasca-statsd:
    image: monasca/statsd:master-20180615-123838
    environment:
      FORWARDER_URL: http://agent-forwarder:17123
      LOG_LEVEL: WARN
    ports:
      - "8125/udp"

[zhangjianweibj]

thanks.i replace docker-compse.yml ${variable} with env value.

[zhangjianweibj]

when monasca-docker_default network created,or which component create this network?

[matrixik]

I believe it's default network created by docker-compose.
It's name is based on your folder name: monasca-docker_default
You are running it from monasca-docker folder.

[zhangjianweibj]

yes,network created by docker default.so i create a network type overlay.then run docker-compose.yml
but components can not access kafka.

[zhangjianweibj]

docker-compose.yml file,network use mynetwork

[zhangjianweibj]

and all components in mynetwork

[zhangjianweibj]

and agent-forward can not access monasca as well

[shail8989]

i am finding the same error where monasca-persistor is keep falling in docker containers . i am running wallaby all in one .

and when i started to checking the logs it

ERROR monasca_persister.repositories.persister r = adapter.send(request, **kwargs)
ERROR monasca_persister.repositories.persister File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
ERROR monasca_persister.repositories.persister raise ConnectionError(err, request=request)
ERROR monasca_persister.repositories.persister requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
ERROR monasca_persister.repositories.persister INFO monasca_persister.persister [-] Received signal 17, beginning graceful shutdown.

my monasca container is not stable at all, any one faced the same issue