Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]按照文档一路安装，从1.2-到1.4【logseer】 真用不起 #167

Open
snpyeso opened this issue Sep 24, 2022 · 2 comments
Open

[Bug]按照文档一路安装，从1.2-到1.4【logseer】 真用不起 #167

snpyeso opened this issue Sep 24, 2022 · 2 comments
Assignees
@ethfoo @zhujf1989 @weilaaa

Comments

@snpyeso
Copy link

snpyeso commented Sep 24, 2022 •
edited

继续观光一下
【Hotplug】hotplugs.hotplug.kubecube.io v1
common 与 pivot-cluster 都打开

spec:
  component:
    -
      name: audit
      status: enabled
    -
      env: "address: elasticsearch-master-headless.elasticsearch.svc\n"
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: enabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: enabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://172.31.0.171:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: enabled
spec:
  component:
    -
      env: "address: elasticsearch-master.elasticsearch.svc \n"
      name: logseer
      status: enabled
    -
      env: "grafana:\n  enabled: true\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://kubecube-thanos-receive:19291/api/v1/receive\n"
      name: kubecube-monitoring
    -
      env: "receive:\n  tsdbRetention: 7d\n  replicaCount: 1\n  replicationFactor: 1\n"
      name: kubecube-thanos
      status: enabled

默认配置，配置 elasticsearch-master-headless.elasticsearch.svc和elasticsearch-master.elasticsearch.svc 都配置过，理论上不会有什么影响，还是不行,然后进行调试

问题一：查询日志报错 “request elasticsearch fail”
问题二：操作审计无数据（经过调试已解决）
过程如下：
查看logseer运行pod的容器日志发现如下

2022-09-24 20:40:47.299 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpress"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "elasticsearch-master.elasticsearch.svc:9200"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1664019350313,
                "lte" : 1664022950313,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1m",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-09-24 20:40:48.302 [http-nio-8080-exec-10]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
	at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

出现一个空指针异常

  • 要么取出ES的数据为空
  • 要么查找拼接的地址为空毕竟有一个没看到访问HOST地址的请求 request to es, url: /*/_search?ignore_unavailable=true
    进入logseer容器

直接curl http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true 发现返回了一大堆数据，证明ES连通性是好的，不过毕竟没添加参数，curl不是很好加参数，加了参数可能就返回空报错了。
猜测是不是环境变量没有设置起，不停的调整环境变量格式，甚至configMap,内部配置文件 期望出现日志
equest to es, url: http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true。
是不是没有读取到address: elasticsearch-master.elasticsearch.svc 这个变量，最终放弃也许日志本来就是这么写的。
转入logagent 的filebeat 的configMap 发现
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:30435]
这个根本访问不到修改成
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:9200]
再试试，嗯一样的不通(好在的是filebeate不爆连接错误了 )
接着看了一下文档也没发现哪里不对，再修复下审计
我本来也安装的内部ES，还是当外部配置下吧

kubectl edit deploy audit -n kubecube-system
env:
- name: AUDIT_WEBHOOK_HOST
  value: http://elasticsearch-master.elasticsearch:9200
- name: AUDIT_WEBHOOK_INDEX
  value: audit
- name: AUDIT_WEBHOOK_TYPE
  value: logs

审计可以了，

但是日志依然不通，看来只有想办法开放ES 9200端口出来用工具连连是没上传还是没查询到，
不过大体定位到如下可能的几个问题

  • 可能是logseer没有读取到环境变量导致出错
  • filebeate配置错误没有上传成功，导致查询空数据（这个数据内容空就报错应该不至于），但是确实直接安装的filebeat有一个connect 错误需要修复
    费解

目前发现的问题猜测ripple和filebeat的配置感觉这里嫌疑最大，创建了新的日志抓取任务，也没看到/etc/filebeat/inputs.d 有什么文件改动
不过也建议修复下空报错的问题，让指示得更明显，只能去看看哪里有源码了
@snpyeso snpyeso changed the title [Bug]按照文档一路安装，从1.2-到1.4, 【audit】【logseer】 这两真用不起 [Bug]按照文档一路安装，从1.2-到1.4【logseer】 这两真用不起 Sep 24, 2022
@snpyeso snpyeso changed the title [Bug]按照文档一路安装，从1.2-到1.4【logseer】 这两真用不起 [Bug]按照文档一路安装，从1.2-到1.4【logseer】 真用不起 Sep 24, 2022
@ethfoo
Copy link

ethfoo commented Oct 24, 2022

应该是logseer读取不到es地址或者es地址无法访问导致的。
logseer里的配置默认应该是一个svc域名，麻烦确认一下：

  1. 域名是否正确（和es的svc对应）
  2. 域名是否被logseer读取
  3. 域名是否可解析
  4. 域名网络是否可达

@snpyeso
Copy link
Author

snpyeso commented Oct 27, 2022

进入logseer容器 直接curl http://elasticsearch-master.elasticsearch.svc:9200/*/_search?ignore_unavailable=true，可有返回数据，是不是证明域名，连通性这些都是OK的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ethfoo ethfoo

@zhujf1989 zhujf1989

@weilaaa weilaaa

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ethfoo @zhujf1989 @snpyeso @weilaaa