Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]日志查询报错,开启logersee后 request elasticsearch fail #126

Open
snpyeso opened this issue Jun 13, 2022 · 11 comments
Open
Assignees

Comments

@snpyeso
Copy link

snpyeso commented Jun 13, 2022

使用日志采集

apiVersion: netease.com/v1
kind: Logconfig
metadata:
  creationTimestamp: '2022-06-13T05:59:27Z'
  generation: 1
  labels:
    app: dep-staff
  managedFields:
    -
      apiVersion: netease.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app': {}
        'f:spec':
          .: {}
          'f:inputs': {}
      manager: Mozilla
      operation: Update
      time: '2022-06-13T05:59:27Z'
  name: stufflog
  namespace: cqdx
  resourceVersion: '50486'
  uid: 4c2ba645-762e-4354-bae4-d9cddd1e18b4
spec:
  inputs:
    -
      enable: true
      type:
        name: dockerStdout

image

image
点击日志查询
image
报错的接口为
image

检查热插拔组件

[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME            PHASE   AGE
common          fail    2d2h
pivot-cluster   fail    2d2h

查看组件

spec:
  component:
    -
      name: audit
      status: disabled
    -
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: disabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: disabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://10.10.10.44:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: disabled
status:
  phase: fail
  results:
    -
      message: 'audit is disabled'
      name: audit
      result: success
      status: disabled
    -
      message: uninstalled
      name: logseer
      result: success
      status: disabled
    -
      message: 'release is running'
      name: logagent
      result: success
      status: enabled
    -
      message: 'release is running'
      name: elasticsearch
      result: success
      status: enabled
    -
      message: 'helm install fail, cannot re-use a name that is still in use'
      name: kubecube-monitoring
      result: fail
      status: enabled
    -
      message: 'release is running'
      name: kubecube-thanos
      result: success
      status: enabled

看了下官方文档也没说如何才能开启日志查询

@tiancandevloper
Copy link
Contributor

https://www.kubecube.io/docs/user-guide/administration/audit/
可以参考这篇文章,将logseer和logagent开启

@zhujf1989
Copy link
Contributor

zhujf1989 commented Jun 16, 2022

可以参考文档:https://www.kubecube.io/docs/installation-guide/enable-plugins/
修改HotPlug的配置, 将logseer及logagent enable

- name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: enabled                            ## 启停标识:这里disabled为禁用
- name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识:这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换

@snpyeso
Copy link
Author

snpyeso commented Jun 16, 2022

可以参考文档:https://www.kubecube.io/docs/installation-guide/enable-plugins/ 修改HotPlug的配置, 将logseer及logagent enable

- name: logseer                               ## 日志管理组件
    namespace: logseer
    pkgName: logseer-v1.0.0.tgz
    status: enabled                            ## 启停标识:这里disabled为禁用
- name: logagent                              ## 日志采集代理组件
    namespace: logagent
    pkgName: logagent-v1.0.0.tgz
    status: enabled                             ## 启停标识:这里enabled为启用
    env: |                                      ## 环境变量
      clustername: "{{.cluster}}"               ## {{.cluster}} 程序会自动注入集群名字替换

这篇文章我看过,但是我的hotplug状态为fail 不知道有没有影响,文档中是runing,看了一会儿也没发现如何让它变成running状态

[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME            PHASE   AGE
common          fail    2d2h
pivot-cluster   fail    2d2h

将logseer及logagent enable 之后

kubectl get pod --all-namespaces
NAMESPACE             NAME                                                      READY   STATUS      RESTARTS       AGE
cqdxdevspace          dep-prepare-6596d79cf-mcmqw                               1/1     Running     0              19h
elasticsearch         elasticsearch-master-0                                    1/1     Running     0              2d
hnc-system            hnc-controller-manager-6d47799b76-kp4wl                   1/1     Running     1 (2d ago)     2d
ingress-nginx         ingress-nginx-admission-create-ggrhg                      0/1     Completed   0              2d
ingress-nginx         ingress-nginx-admission-patch-chxhk                       0/1     Completed   1              2d
ingress-nginx         ingress-nginx-controller-776d88d5c9-mthdn                 1/1     Running     0              2d
kube-system           calico-kube-controllers-d84f7b88b-krrjs                   1/1     Running     0              2d
kube-system           calico-node-b55ds                                         1/1     Running     0              2d
kube-system           coredns-65c54cc984-l4tpt                                  1/1     Running     0              2d
kube-system           coredns-65c54cc984-rq6b8                                  1/1     Running     0              2d
kube-system           etcd-localhost.localdomain                                1/1     Running     8              2d
kube-system           kube-apiserver-localhost.localdomain                      1/1     Running     0              26h
kube-system           kube-controller-manager-localhost.localdomain             1/1     Running     28 (26h ago)   2d
kube-system           kube-proxy-7bgj5                                          1/1     Running     0              2d
kube-system           kube-scheduler-localhost.localdomain                      1/1     Running     22 (26h ago)   2d
kube-system           metrics-server-6d69ff96c6-7s27b                           1/1     Running     3 (26h ago)    2d
kubecube-monitoring   kubecube-monitoring-kube-state-metrics-6cf5c8c6d4-dqv2t   1/1     Running     0              2d
kubecube-monitoring   kubecube-monitoring-operator-5f84949849-sjtwq             1/1     Running     0              2d
kubecube-monitoring   kubecube-monitoring-prometheus-node-exporter-c9cdt        1/1     Running     0              2d
kubecube-monitoring   kubecube-thanos-query-86f669566f-ndwm7                    1/1     Running     0              2d
kubecube-monitoring   kubecube-thanos-receive-0                                 1/1     Running     0              2d
kubecube-monitoring   prometheus-kubecube-monitoring-prometheus-0               2/2     Running     0              2d
kubecube-system       audit-78b7d4cf7d-4h2hq                                    1/1     Running     0              2d
kubecube-system       cloud-shell-5c87948759-4zcwq                              1/1     Running     0              2d
kubecube-system       frontend-7cbccf84fb-vh5hv                                 1/1     Running     0              2d
kubecube-system       kubecube-78b59d768d-tpssk                                 1/1     Running     3 (26h ago)    2d
kubecube-system       warden-d876d9fd8-969q5                                    1/1     Running     0              2d
kubecube-system       webconsole-7d7f989996-wvh2r                               1/1     Running     3 (26h ago)    2d
local-path-storage    local-path-provisioner-86b7fd94b5-gh2wl                   1/1     Running     3 (26h ago)    2d
logagent              filebeat-dhwr5                                            2/2     Running     0              6m16s

没有发现 logseer相关的pod

查看common yaml 发现logseer 为uninstalled

spec:
  component:
    -
      name: audit
      status: disabled
    -
      name: logseer
      namespace: logseer
      pkgName: logseer-v1.0.0.tgz
      status: enabled
    -
      env: "clustername: \"{{.cluster}}\"\n"
      name: logagent
      namespace: logagent
      pkgName: logagent-v1.0.0.tgz
      status: enabled
    -
      name: elasticsearch
      namespace: elasticsearch
      pkgName: elasticsearch-7.8.1.tgz
      status: enabled
    -
      env: "grafana:\n  enabled: false\nprometheus:\n  prometheusSpec:\n    externalLabels:\n      cluster: \"{{.cluster}}\"\n    remoteWrite:\n    - url: http://10.10.10.44:31291/api/v1/receive\n"
      name: kubecube-monitoring
      namespace: kubecube-monitoring
      pkgName: kubecube-monitoring-15.4.12.tgz
      status: enabled
    -
      name: kubecube-thanos
      namespace: kubecube-monitoring
      pkgName: thanos-3.18.0.tgz
      status: disabled
status:
  phase: fail
  results:
    -
      message: 'audit is disabled'
      name: audit
      result: success
      status: disabled
    -
      message: uninstalled
      name: logseer
      result: success
      status: disabled
    -
      message: 'release is running'
      name: logagent
      result: success
      status: enabled
    -
      message: 'release is running'
      name: elasticsearch
      result: success
      status: enabled
    -
      message: 'helm install fail, cannot re-use a name that is still in use'
      name: kubecube-monitoring
      result: fail
      status: enabled
    -
      message: 'release is running'
      name: kubecube-thanos
      result: success
      status: enabled

检查下发现 有common和pivot-cluster 的热插拔,发现common 和 pivot-cluster都有 logseer配置,小白都先开启吧,然后再开启elasticsearch,
呃…… 这次错误提示还是蛮明显的,但是不知道如何修复
{"reason":400,"code":400,"message":"request elasticsearch fail"}
image
检查下elasticsearch的服务 发现一切正常

[root@localhost ~]# kubectl get all -n elasticsearch
NAME                         READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0   1/1     Running   0          147m

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch-master            ClusterIP   10.108.228.229   <none>        9200/TCP,9300/TCP   147m
service/elasticsearch-master-headless   ClusterIP   None             <none>        9200/TCP,9300/TCP   147m

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   1/1     147m

@zhujf1989
Copy link
Contributor

/cc @weilaaa

@snpyeso snpyeso changed the title [Bug]日志查询报错,是否需要单独开启logersee么 [Bug]日志查询报错,是否需要单独开启logersee后 request elasticsearch fail Jun 28, 2022
@snpyeso snpyeso changed the title [Bug]日志查询报错,是否需要单独开启logersee后 request elasticsearch fail [Bug]日志查询报错,开启logersee后 request elasticsearch fail Jun 28, 2022
@snpyeso
Copy link
Author

snpyeso commented Jun 30, 2022

重新安装kubecube,使用的最新版,全部开启common里的插件和pivot-cluster里的插件,进入日志查询,依然 request elasticsearch fail
安装结果也和上面差不多

@snpyeso
Copy link
Author

snpyeso commented Jul 7, 2022

@ethfoo
Copy link

ethfoo commented Jul 12, 2022

麻烦发下请求的时候logseer的日志

@snpyeso
Copy link
Author

snpyeso commented Jul 13, 2022

麻烦发下请求的时候logseer的日志

好的稍等,我得重装一下,因为我尝试自己修改已经有点改得记不到哪里要恢复了

@snpyeso
Copy link
Author

snpyeso commented Jul 13, 2022

@ethfoo
安装了两台机器

[root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME                              STATUS   ROLES                  AGE    VERSION
zpfrltgup4tujpi1-0001.novalocal   Ready    control-plane,master   7m3s   v1.23.5
zpfrltgup4tujpi1.novalocal        Ready    node                   32s    v1.23.5

进入空间配置好基本环境 新增租户空间等

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME                         STATUS   AGE
default                      Active   13m
elasticsearch                Active   12m
hnc-system                   Active   12m
ingress-nginx                Active   12m
kube-node-lease              Active   13m
kube-public                  Active   13m
kube-system                  Active   13m
kubecube-monitoring          Active   13m
kubecube-project-wordpress   Active   113s
kubecube-system              Active   12m
kubecube-tenant-tenant       Active   2m2s
local-path-storage           Active   12m
wordpresspace                Active   50s

添加一个nginx 测试deployment

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n  wordpresspace
NAME                          READY   STATUS    RESTARTS   AGE
pod/dep-ng-5f6b6f4778-j5vb2   1/1     Running   0          62s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dep-ng   1/1     1            1           62s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/dep-ng-5f6b6f4778   1         1         1       62s

开启全部热插件

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name:         common
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:13Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:01Z
  Resource Version:  6058
  UID:               fc94d067-3712-4907-aca6-48d7f4095314
Spec:
  Component:
    Name:       audit
    Status:     enabled
    Name:       logseer
    Namespace:  logseer
    Pkg Name:   logseer-v1.0.0.tgz
    Status:     enabled
    Env:        clustername: "{{.cluster}}"

    Name:       logagent
    Namespace:  logagent
    Pkg Name:   logagent-v1.0.0.tgz
    Status:     enabled
    Name:       elasticsearch
    Namespace:  elasticsearch
    Pkg Name:   elasticsearch-7.8.1.tgz
    Status:     enabled
    Env:        grafana:
  enabled: false
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://172.31.0.171:31291/api/v1/receive

    Name:       kubecube-monitoring
    Namespace:  kubecube-monitoring
    Pkg Name:   kubecube-monitoring-15.4.12.tgz
    Status:     enabled
    Name:       kubecube-thanos
    Namespace:  kubecube-monitoring
    Pkg Name:   thanos-3.18.0.tgz
    Status:     enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled
Events:       <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name:         pivot-cluster
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:15Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:09Z
  Resource Version:  6066
  UID:               098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
  Component:
    Name:    logseer
    Status:  enabled
    Env:     grafana:
  enabled: true
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://kubecube-thanos-receive:19291/api/v1/receive

    Name:  kubecube-monitoring
    Env:   receive:
  tsdbRetention: 7d
  replicaCount: 1
  replicationFactor: 1

    Name:    kubecube-thanos
    Status:  enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled

配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng

image

点击日志查询
image

查看logseer日志

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer 
NAME                           READY   STATUS    RESTARTS   AGE
pod/logseer-657645d857-w5rv5   1/1     Running   0          6m11s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/logseer   ClusterIP   10.105.118.70   <none>        80/TCP    6m11s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/logseer   1/1     1            1           6m11s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/logseer-657645d857   1         1         1       6m11s

##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpresspace"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "*"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1657675115512,
                "lte" : 1657678715512,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1d",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-07-13 10:22:12.384 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
	at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

@snpyeso
Copy link
Author

snpyeso commented Aug 2, 2022

@ethfoo @zhujf1989 @tiancandevloper @hzliangbin 可以看一看么

@snpyeso
Copy link
Author

snpyeso commented Aug 23, 2022

@ethfoo 安装了两台机器

[root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME                              STATUS   ROLES                  AGE    VERSION
zpfrltgup4tujpi1-0001.novalocal   Ready    control-plane,master   7m3s   v1.23.5
zpfrltgup4tujpi1.novalocal        Ready    node                   32s    v1.23.5

进入空间配置好基本环境 新增租户空间等

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME                         STATUS   AGE
default                      Active   13m
elasticsearch                Active   12m
hnc-system                   Active   12m
ingress-nginx                Active   12m
kube-node-lease              Active   13m
kube-public                  Active   13m
kube-system                  Active   13m
kubecube-monitoring          Active   13m
kubecube-project-wordpress   Active   113s
kubecube-system              Active   12m
kubecube-tenant-tenant       Active   2m2s
local-path-storage           Active   12m
wordpresspace                Active   50s

添加一个nginx 测试deployment

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n  wordpresspace
NAME                          READY   STATUS    RESTARTS   AGE
pod/dep-ng-5f6b6f4778-j5vb2   1/1     Running   0          62s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dep-ng   1/1     1            1           62s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/dep-ng-5f6b6f4778   1         1         1       62s

开启全部热插件

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name:         common
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:13Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:01Z
  Resource Version:  6058
  UID:               fc94d067-3712-4907-aca6-48d7f4095314
Spec:
  Component:
    Name:       audit
    Status:     enabled
    Name:       logseer
    Namespace:  logseer
    Pkg Name:   logseer-v1.0.0.tgz
    Status:     enabled
    Env:        clustername: "{{.cluster}}"

    Name:       logagent
    Namespace:  logagent
    Pkg Name:   logagent-v1.0.0.tgz
    Status:     enabled
    Name:       elasticsearch
    Namespace:  elasticsearch
    Pkg Name:   elasticsearch-7.8.1.tgz
    Status:     enabled
    Env:        grafana:
  enabled: false
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://172.31.0.171:31291/api/v1/receive

    Name:       kubecube-monitoring
    Namespace:  kubecube-monitoring
    Pkg Name:   kubecube-monitoring-15.4.12.tgz
    Status:     enabled
    Name:       kubecube-thanos
    Namespace:  kubecube-monitoring
    Pkg Name:   thanos-3.18.0.tgz
    Status:     enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled
Events:       <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name:         pivot-cluster
Namespace:    
Labels:       <none>
Annotations:  kubecube.io/sync: true
API Version:  hotplug.kubecube.io/v1
Kind:         Hotplug
Metadata:
  Creation Timestamp:  2022-07-13T01:57:47Z
  Generation:          2
  Managed Fields:
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubecube.io/sync:
      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-13T01:57:47Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:phase:
        f:results:
    Manager:      warden
    Operation:    Update
    Subresource:  status
    Time:         2022-07-13T02:04:15Z
    API Version:  hotplug.kubecube.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:component:
    Manager:         Mozilla
    Operation:       Update
    Time:            2022-07-13T02:14:09Z
  Resource Version:  6066
  UID:               098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
  Component:
    Name:    logseer
    Status:  enabled
    Env:     grafana:
  enabled: true
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: "{{.cluster}}"
    remoteWrite:
    - url: http://kubecube-thanos-receive:19291/api/v1/receive

    Name:  kubecube-monitoring
    Env:   receive:
  tsdbRetention: 7d
  replicaCount: 1
  replicationFactor: 1

    Name:    kubecube-thanos
    Status:  enabled
Status:
  Phase:  fail
  Results:
    Message:  audit is enabled
    Name:     audit
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logseer
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     logagent
    Result:   success
    Status:   enabled
    Message:  release is running
    Name:     elasticsearch
    Result:   success
    Status:   enabled
    Message:  helm install fail, cannot re-use a name that is still in use
    Name:     kubecube-monitoring
    Result:   fail
    Status:   enabled
    Message:  release is running
    Name:     kubecube-thanos
    Result:   success
    Status:   enabled

配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng

image

点击日志查询 image

查看logseer日志

[root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer 
NAME                           READY   STATUS    RESTARTS   AGE
pod/logseer-657645d857-w5rv5   1/1     Running   0          6m11s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/logseer   ClusterIP   10.105.118.70   <none>        80/TCP    6m11s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/logseer   1/1     1            1           6m11s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/logseer-657645d857   1         1         1       6m11s

##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:52   INFO  - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
    "size": 50,
    "from": 0,
    "query": {
      "bool" : {
        "filter" : [
            {"term": {"cluster_name" : "pivot-cluster"}},
            {"term": {"namespace" : "wordpresspace"}}
        ],
        "must" : [
          {
            "query_string" : {
              "default_field" : "message",
              "query" : "*"
            }
          },
          {
            "range" : {
              "@timestamp" : {
                "gte" : 1657675115512,
                "lte" : 1657678715512,
                "format": "epoch_millis"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1d",
          "time_zone": "Asia/Shanghai",
          "min_doc_count": 1
        }
      }
    },
    "highlight" : {
      "fields" : {
        "message" : {}
      },
      "fragment_size": 2147483647
    },
    "sort" : [
      { "@timestamp" : "asc"}
    ],
    "_source" : {
      "excludes": "tags"
    },
    "timeout": "30000ms"
} 
2022-07-13 10:22:12.384 [http-nio-8080-exec-5]    c.n.logseer.engine.impl.ElasticSearchEngineImpl:65   ERROR - request elasticsearch exception: {} 
java.net.ConnectException: null
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
	at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
	at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

期待回复

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants