-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]日志查询报错,开启logersee后 request elasticsearch fail #126
Comments
https://www.kubecube.io/docs/user-guide/administration/audit/ |
可以参考文档:https://www.kubecube.io/docs/installation-guide/enable-plugins/
|
这篇文章我看过,但是我的hotplug状态为fail 不知道有没有影响,文档中是runing,看了一会儿也没发现如何让它变成running状态 [root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug
NAME PHASE AGE
common fail 2d2h
pivot-cluster fail 2d2h 将logseer及logagent enable 之后 kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
cqdxdevspace dep-prepare-6596d79cf-mcmqw 1/1 Running 0 19h
elasticsearch elasticsearch-master-0 1/1 Running 0 2d
hnc-system hnc-controller-manager-6d47799b76-kp4wl 1/1 Running 1 (2d ago) 2d
ingress-nginx ingress-nginx-admission-create-ggrhg 0/1 Completed 0 2d
ingress-nginx ingress-nginx-admission-patch-chxhk 0/1 Completed 1 2d
ingress-nginx ingress-nginx-controller-776d88d5c9-mthdn 1/1 Running 0 2d
kube-system calico-kube-controllers-d84f7b88b-krrjs 1/1 Running 0 2d
kube-system calico-node-b55ds 1/1 Running 0 2d
kube-system coredns-65c54cc984-l4tpt 1/1 Running 0 2d
kube-system coredns-65c54cc984-rq6b8 1/1 Running 0 2d
kube-system etcd-localhost.localdomain 1/1 Running 8 2d
kube-system kube-apiserver-localhost.localdomain 1/1 Running 0 26h
kube-system kube-controller-manager-localhost.localdomain 1/1 Running 28 (26h ago) 2d
kube-system kube-proxy-7bgj5 1/1 Running 0 2d
kube-system kube-scheduler-localhost.localdomain 1/1 Running 22 (26h ago) 2d
kube-system metrics-server-6d69ff96c6-7s27b 1/1 Running 3 (26h ago) 2d
kubecube-monitoring kubecube-monitoring-kube-state-metrics-6cf5c8c6d4-dqv2t 1/1 Running 0 2d
kubecube-monitoring kubecube-monitoring-operator-5f84949849-sjtwq 1/1 Running 0 2d
kubecube-monitoring kubecube-monitoring-prometheus-node-exporter-c9cdt 1/1 Running 0 2d
kubecube-monitoring kubecube-thanos-query-86f669566f-ndwm7 1/1 Running 0 2d
kubecube-monitoring kubecube-thanos-receive-0 1/1 Running 0 2d
kubecube-monitoring prometheus-kubecube-monitoring-prometheus-0 2/2 Running 0 2d
kubecube-system audit-78b7d4cf7d-4h2hq 1/1 Running 0 2d
kubecube-system cloud-shell-5c87948759-4zcwq 1/1 Running 0 2d
kubecube-system frontend-7cbccf84fb-vh5hv 1/1 Running 0 2d
kubecube-system kubecube-78b59d768d-tpssk 1/1 Running 3 (26h ago) 2d
kubecube-system warden-d876d9fd8-969q5 1/1 Running 0 2d
kubecube-system webconsole-7d7f989996-wvh2r 1/1 Running 3 (26h ago) 2d
local-path-storage local-path-provisioner-86b7fd94b5-gh2wl 1/1 Running 3 (26h ago) 2d
logagent filebeat-dhwr5 2/2 Running 0 6m16s
没有发现 logseer相关的pod 查看common yaml 发现logseer 为uninstalled spec:
component:
-
name: audit
status: disabled
-
name: logseer
namespace: logseer
pkgName: logseer-v1.0.0.tgz
status: enabled
-
env: "clustername: \"{{.cluster}}\"\n"
name: logagent
namespace: logagent
pkgName: logagent-v1.0.0.tgz
status: enabled
-
name: elasticsearch
namespace: elasticsearch
pkgName: elasticsearch-7.8.1.tgz
status: enabled
-
env: "grafana:\n enabled: false\nprometheus:\n prometheusSpec:\n externalLabels:\n cluster: \"{{.cluster}}\"\n remoteWrite:\n - url: http://10.10.10.44:31291/api/v1/receive\n"
name: kubecube-monitoring
namespace: kubecube-monitoring
pkgName: kubecube-monitoring-15.4.12.tgz
status: enabled
-
name: kubecube-thanos
namespace: kubecube-monitoring
pkgName: thanos-3.18.0.tgz
status: disabled
status:
phase: fail
results:
-
message: 'audit is disabled'
name: audit
result: success
status: disabled
-
message: uninstalled
name: logseer
result: success
status: disabled
-
message: 'release is running'
name: logagent
result: success
status: enabled
-
message: 'release is running'
name: elasticsearch
result: success
status: enabled
-
message: 'helm install fail, cannot re-use a name that is still in use'
name: kubecube-monitoring
result: fail
status: enabled
-
message: 'release is running'
name: kubecube-thanos
result: success
status: enabled 检查下发现 有common和pivot-cluster 的热插拔,发现common 和 pivot-cluster都有 logseer配置,小白都先开启吧,然后再开启elasticsearch, [root@localhost ~]# kubectl get all -n elasticsearch
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 147m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.108.228.229 <none> 9200/TCP,9300/TCP 147m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 147m
NAME READY AGE
statefulset.apps/elasticsearch-master 1/1 147m |
/cc @weilaaa |
重新安装kubecube,使用的最新版,全部开启common里的插件和pivot-cluster里的插件,进入日志查询,依然 request elasticsearch fail |
麻烦发下请求的时候logseer的日志 |
好的稍等,我得重装一下,因为我尝试自己修改已经有点改得记不到哪里要恢复了 |
@ethfoo [root@zpfrltgup4tujpi1 manifests]# kubectl get node
NAME STATUS ROLES AGE VERSION
zpfrltgup4tujpi1-0001.novalocal Ready control-plane,master 7m3s v1.23.5
zpfrltgup4tujpi1.novalocal Ready node 32s v1.23.5 进入空间配置好基本环境 新增租户空间等 [root@zpfrltgup4tujpi1-0001 manifests]# kubectl get ns
NAME STATUS AGE
default Active 13m
elasticsearch Active 12m
hnc-system Active 12m
ingress-nginx Active 12m
kube-node-lease Active 13m
kube-public Active 13m
kube-system Active 13m
kubecube-monitoring Active 13m
kubecube-project-wordpress Active 113s
kubecube-system Active 12m
kubecube-tenant-tenant Active 2m2s
local-path-storage Active 12m
wordpresspace Active 50s 添加一个nginx 测试deployment [root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n wordpresspace
NAME READY STATUS RESTARTS AGE
pod/dep-ng-5f6b6f4778-j5vb2 1/1 Running 0 62s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dep-ng 1/1 1 1 62s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dep-ng-5f6b6f4778 1 1 1 62s 开启全部热插件 [root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug common
Name: common
Namespace:
Labels: <none>
Annotations: kubecube.io/sync: true
API Version: hotplug.kubecube.io/v1
Kind: Hotplug
Metadata:
Creation Timestamp: 2022-07-13T01:57:47Z
Generation: 2
Managed Fields:
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kubecube.io/sync:
f:spec:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-13T01:57:47Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:phase:
f:results:
Manager: warden
Operation: Update
Subresource: status
Time: 2022-07-13T02:04:13Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:component:
Manager: Mozilla
Operation: Update
Time: 2022-07-13T02:14:01Z
Resource Version: 6058
UID: fc94d067-3712-4907-aca6-48d7f4095314
Spec:
Component:
Name: audit
Status: enabled
Name: logseer
Namespace: logseer
Pkg Name: logseer-v1.0.0.tgz
Status: enabled
Env: clustername: "{{.cluster}}"
Name: logagent
Namespace: logagent
Pkg Name: logagent-v1.0.0.tgz
Status: enabled
Name: elasticsearch
Namespace: elasticsearch
Pkg Name: elasticsearch-7.8.1.tgz
Status: enabled
Env: grafana:
enabled: false
prometheus:
prometheusSpec:
externalLabels:
cluster: "{{.cluster}}"
remoteWrite:
- url: http://172.31.0.171:31291/api/v1/receive
Name: kubecube-monitoring
Namespace: kubecube-monitoring
Pkg Name: kubecube-monitoring-15.4.12.tgz
Status: enabled
Name: kubecube-thanos
Namespace: kubecube-monitoring
Pkg Name: thanos-3.18.0.tgz
Status: enabled
Status:
Phase: fail
Results:
Message: audit is enabled
Name: audit
Result: success
Status: enabled
Message: release is running
Name: logseer
Result: success
Status: enabled
Message: release is running
Name: logagent
Result: success
Status: enabled
Message: release is running
Name: elasticsearch
Result: success
Status: enabled
Message: helm install fail, cannot re-use a name that is still in use
Name: kubecube-monitoring
Result: fail
Status: enabled
Message: release is running
Name: kubecube-thanos
Result: success
Status: enabled
Events: <none>
[root@zpfrltgup4tujpi1-0001 manifests]# kubectl describe hotplug pivot-cluster
Name: pivot-cluster
Namespace:
Labels: <none>
Annotations: kubecube.io/sync: true
API Version: hotplug.kubecube.io/v1
Kind: Hotplug
Metadata:
Creation Timestamp: 2022-07-13T01:57:47Z
Generation: 2
Managed Fields:
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:kubecube.io/sync:
f:spec:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-13T01:57:47Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:phase:
f:results:
Manager: warden
Operation: Update
Subresource: status
Time: 2022-07-13T02:04:15Z
API Version: hotplug.kubecube.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:component:
Manager: Mozilla
Operation: Update
Time: 2022-07-13T02:14:09Z
Resource Version: 6066
UID: 098fe080-bbc9-40db-a7c6-b8e36290191e
Spec:
Component:
Name: logseer
Status: enabled
Env: grafana:
enabled: true
prometheus:
prometheusSpec:
externalLabels:
cluster: "{{.cluster}}"
remoteWrite:
- url: http://kubecube-thanos-receive:19291/api/v1/receive
Name: kubecube-monitoring
Env: receive:
tsdbRetention: 7d
replicaCount: 1
replicationFactor: 1
Name: kubecube-thanos
Status: enabled
Status:
Phase: fail
Results:
Message: audit is enabled
Name: audit
Result: success
Status: enabled
Message: release is running
Name: logseer
Result: success
Status: enabled
Message: release is running
Name: logagent
Result: success
Status: enabled
Message: release is running
Name: elasticsearch
Result: success
Status: enabled
Message: helm install fail, cannot re-use a name that is still in use
Name: kubecube-monitoring
Result: fail
Status: enabled
Message: release is running
Name: kubecube-thanos
Result: success
Status: enabled
配置日志任务管理添加为容器日志标准输出 标签为 kubecube.io/app: dep-ng 查看logseer日志 [root@zpfrltgup4tujpi1-0001 manifests]# kubectl get all -n logseer
NAME READY STATUS RESTARTS AGE
pod/logseer-657645d857-w5rv5 1/1 Running 0 6m11s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/logseer ClusterIP 10.105.118.70 <none> 80/TCP 6m11s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/logseer 1/1 1 1 6m11s
NAME DESIRED CURRENT READY AGE
replicaset.apps/logseer-657645d857 1 1 1 6m11s
##日志如下
2022-07-13 10:22:11.377 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:52 INFO - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
"size": 50,
"from": 0,
"query": {
"bool" : {
"filter" : [
{"term": {"cluster_name" : "pivot-cluster"}},
{"term": {"namespace" : "wordpresspace"}}
],
"must" : [
{
"query_string" : {
"default_field" : "message",
"query" : "*"
}
},
{
"range" : {
"@timestamp" : {
"gte" : 1657675115512,
"lte" : 1657678715512,
"format": "epoch_millis"
}
}
}
]
}
},
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "1d",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
}
}
},
"highlight" : {
"fields" : {
"message" : {}
},
"fragment_size": 2147483647
},
"sort" : [
{ "@timestamp" : "asc"}
],
"_source" : {
"excludes": "tags"
},
"timeout": "30000ms"
}
2022-07-13 10:22:12.384 [http-nio-8080-exec-5] c.n.logseer.engine.impl.ElasticSearchEngineImpl:65 ERROR - request elasticsearch exception: {}
java.net.ConnectException: null
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
|
期待回复 |
使用日志采集
点击日志查询
报错的接口为
检查热插拔组件
[root@zpfrltgup4tujpi1-0001 network-scripts]# kubectl get hotplug NAME PHASE AGE common fail 2d2h pivot-cluster fail 2d2h
查看组件
看了下官方文档也没说如何才能开启日志查询
The text was updated successfully, but these errors were encountered: