Telemetry daemon silently stops when incorrect messages sent to listening port

[sgaragan]

Description
We had pmacct configured for YANG push telemetry for our Cisco devices but when we enabled telemetry from our Juniper boxes, which use gRPC, pmacct stopped accepting all YANG telemetry. It did this silently, putting out a single log file message showing data coming from Juniper and then nothing for the YANG telemetry after that. Once we disabled the telemetry from Juniper, the telemetry daemon started to process messages again, without having to restart the server

It looks like there needs to be additional error handling for the telemetry daemon to output error messages when it gets data it cannot understand and still allow other data to be processed.

Our concern is that if there is one device that sends messages that cause the telemetry daemon to choke, we could lose all message processing silently

The telemetry config is:

telemetry_daemon: true
telemetry_daemon_decoder: cisco_v1
telemetry_daemon_port_tcp: 10005
telemetry_daemon_max_peers: 800
tmp_telemetry_decode_cisco_v1_json_string: True
telemetry_daemon_tag_map: /etc/pmacct/pretag.map
telemetry_daemon_msglog_kafka_topic: matterhorn.ietf.device-json-raw
telemetry_daemon_msglog_kafka_config_file: /etc/pmacct/librdkafka.conf

Version
1.7.10

Thanks,
Sean

[paololucente]

Hi Sean ( @sgaragan ), of course i see the issue with such a behavior. Thing being, in order to improve the current behavior i would need to somehow reproduce the problem. Is this some sort of lab / test instance that i could potentially access for some troubleshooting / debugging? If so, let's get in touch via unicast email. Paolo