Druid datasource showing 0.0% availability in OCI

[prashanthShiksha]

We're currently encountering an issue with OCI Druid. The datasource is displaying 0.0% availability, indicating a potential problem with the service. Upon investigating further, we found error logs within the Druid services.

• Historical logs indicate a failure to load segment files containing information about segments.
• In the coordinator logs, there's a tier named "_default_tier" that doesn't have any servers associated with it.
  Here is the link of slack channel : https://ekstep.slack.com/archives/C05LC3YDTKJ/p1713158371132489?thread_ts=1711949803.877849&cid=C05LC3YDTKJ

[prashanthShiksha]

[prashanthShiksha]

@vinukumar-vs , @ashokreddy1208,@vijiurs,@aks30,@kiranharidas187,@Preethi-Sl,@gagannani88,@Vivek-M-08,@sowmya-dixit

[sowmya-dixit]

@prashanthShiksha - Can you check for errors under middle manager & overlord Druid services?

[prashanthShiksha]

@sowmya-dixit druid historical service is running but it showing some error i have attached logs here ,
historical.log
could you please check that ?

[prashanthShiksha]

any updates @sowmya-dixit ?

[sowmya-dixit]

@prashanthShiksha - When we checked from the historical logs, its failing to load segments data as few of the segment files are empty. Looks like segment files are corrupted. Was there any manual changes done to Postgres DB/Cloud storage directly?

[prashanthShiksha]

@sowmya-dixit their is no any manual changes done to Postgres DB/Cloud storage from our end .

[prashanthShiksha]

@sowmya-dixit, this is one of the segment files failing to load. Is it loading from the local VM directory? This file is not present in the directory /var/segmentstore0/info_dir/.

"2024-04-18T11:21:46,010 ERROR [main] org.apache.druid.emitter.graphite.GraphiteEmitter - The following alert is dropped, description is [Failed to load segment from segmentInfo file], severity is [component-failure]
""2024-04-18T11:21:46,012 ERROR [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment from segmentInfo file: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class com.fasterxml.jackson.databind.exc.MismatchedInputException, exceptionMessage=No content to map due to end-of-input
at [Source: (File); line: 1, column: 0], file=/var/segmentstore0/info_dir/ml-project-status_2024-04-12T00:00:00.000Z_2024-04-13T00:00:00.000Z_2024-04-12T18:40:49.804Z_19}
" com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input

[sowmya-dixit]

@prashanthShiksha - Historical service was having OOM errors. Its fixed now. All the datasources are available. Please check.

@gandham-santhosh
However do we need so many datasources added in Druid OCI environment? Historical service is having OOM even with 2GB. We are having a small instance for Druid currently, if we are planning to have more data & datasources, we would suggest to use instance with slightly higher configurations.

[prashanthShiksha]

Hi @sowmya-dixit , how did you determine that the Historical service was encountering an Out Of Memory (OOM) error? The historical logs don't seem to contain any messages about OOM. Could you please explain how you resolved this error? This information will be valuable for us to troubleshoot similar issues in the future.

[sowmya-dixit]

@prashanthShiksha - Sure, All the druid services are run as systemd services. We have these OOM error logs under /var/log/syslog.
Below are the historical logs with OOM and restarts details -

Apr 18 14:35:52 druid systemd[1]: Stopped druid_historical_service.
Apr 18 14:35:52 druid systemd[1]: Started druid_historical_service.
Apr 18 14:37:45 druid sh[8771]: java.lang.OutOfMemoryError: Java heap space
Apr 18 14:37:45 druid sh[8771]: Dumping heap to /var/log/druid/crash_logs/historical.hprof ...
Apr 18 14:37:45 druid sh[8771]: Unable to create /var/log/druid/crash_logs/historical.hprof: File exists
Apr 18 14:37:45 druid sh[8771]: Terminating due to java.lang.OutOfMemoryError: Java heap space
Apr 18 14:37:45 druid systemd[1]: druid_historical.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 18 14:37:45 druid systemd[1]: druid_historical.service: Failed with result 'exit-code'.
Apr 18 14:37:45 druid systemd[1]: druid_historical.service: Service hold-off time over, scheduling restart.
Apr 18 14:37:45 druid systemd[1]: druid_historical.service: Scheduled restart job, restart counter is at 41.
Apr 18 14:37:45 druid systemd[1]: Stopped druid_historical_service.
Apr 18 14:37:45 druid systemd[1]: Started druid_historical_service.
Apr 18 14:39:35 druid sh[8982]: java.lang.OutOfMemoryError: Java heap space
Apr 18 14:39:35 druid sh[8982]: Dumping heap to /var/log/druid/crash_logs/historical.hprof ...
Apr 18 14:39:35 druid sh[8982]: Unable to create /var/log/druid/crash_logs/historical.hprof: File exists
Apr 18 14:39:35 druid sh[8982]: Terminating due to java.lang.OutOfMemoryError: Java heap space
Apr 18 14:39:35 druid systemd[1]: druid_historical.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 18 14:39:35 druid systemd[1]: druid_historical.service: Failed with result 'exit-code'.
Apr 18 14:39:35 druid systemd[1]: druid_historical.service: Service hold-off time over, scheduling restart.
Apr 18 14:39:35 druid systemd[1]: druid_historical.service: Scheduled restart job, restart counter is at 42.
Apr 18 14:39:35 druid systemd[1]: Stopped druid_historical_service.
Apr 18 14:39:35 druid systemd[1]: Started druid_historical_service.