Problems trying to integrate the clair notifier

[SuperSandro2000]

Description of Problem / Feature Request

I am trying to integrate the clair notifer API into https://github.com/sapcc/keppel in this PR sapcc/keppel#192.

I have added an endpoint to keppel where clair can send it's webhook.
When a webhook is received, keppel is calling the callback URL and retrieving all paged notifcations, acting upon the notifcation and then deleting them.

In the clair log I saw the following log lines related to the notifier API.

2023/04/18 10:08:12 REQUEST: 147.204.35.163 - - "POST /clair-notification HTTP/1.1" 200 0 "-" "clair/4.6.1 (user)" 0.098s
2023/04/18 10:08:12 ERROR: error while trying to delete notification: cannot DELETE https://clair.example.com/notifier/api/v1/notification/880b4893-6169-4496-b87a-a2cce5f051ae: got 500 resp
onse: "{\"code\":\"internal-error\",\"message\":\"could not delete notification: no receipt exists for notification id 880b4893-6169-4496-b87a-a2cce5f051ae\"}"
{"level":"warn","request_id":"f3e6ea9c9d5cde63","component":"httptransport/NotificationV1.delete","error":"no receipt exists for notification id 880b4893-6169-4496-b87a-a2cce5f051ae","time"
:"2023-04-18T10:08:12Z","message":"could not delete notification"}
{"level":"info","component":"httptransport/New","request_id":"f3e6ea9c9d5cde63","remote_addr":"10.100.4.233:33892","method":"DELETE","request_uri":"/notifier/api/v1/notification/880b4893-61
69-4496-b87a-a2cce5f051ae","status":500,"duration":2.534393,"time":"2023-04-18T10:08:12Z","message":"handled HTTP request"}
{"level":"info","component":"httptransport/New","request_id":"4a69c608a946f218","remote_addr":"10.100.4.233:33892","method":"GET","request_uri":"/notifier/api/v1/notification/234162ad-1b39-
4cee-94ba-1d0858318763","status":200,"duration":17.363467,"time":"2023-04-18T10:08:12Z","message":"handled HTTP request"}
{"level":"info","deliverer":"webhook","component":"notifier/webhook/Deliverer.Deliver","notification_id":"234162ad-1b39-4cee-94ba-1d0858318763","callback":"https://clair.example.com/notifie
r/api/v1/notification/234162ad-1b39-4cee-94ba-1d0858318763","target":"https://keppel.example.com/clair-notification","time":"2023-04-18T10:08:12Z","message":"dispatching webhook"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","component":"notifier/Delivery.RunDelivery","deliverer":"webhook","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}
{"level":"info","deliverer":"webhook","component":"notifier/Delivery.RunDelivery","created":1,"time":"2023-04-18T10:08:12Z","message":"notification ids in created status"}

My question is especially about the following three lines:

2023/04/18 10:08:12 ERROR: error while trying to delete notification: cannot DELETE https://clair.example.com/notifier/api/v1/notification/880b4893-6169-4496-b87a-a2cce5f051ae: got 500 response: "{\"code\":\"internal-error\",\"message\":\"could not delete notification: no receipt exists for notification id 880b4893-6169-4496-b87a-a2cce5f051ae\"}"
{"level":"warn","request_id":"f3e6ea9c9d5cde63","component":"httptransport/NotificationV1.delete","error":"no receipt exists for notification id 880b4893-6169-4496-b87a-a2cce5f051ae","time":"2023-04-18T10:08:12Z","message":"could not delete notification"}
{"level":"info","component":"httptransport/New","request_id":"f3e6ea9c9d5cde63","remote_addr":"10.100.4.233:33892","method":"DELETE","request_uri":"/notifier/api/v1/notification/880b4893-6169-4496-b87a-a2cce5f051ae","status":500,"duration":2.534393,"time":"2023-04-18T10:08:12Z","message":"handled HTTP request"}

As seen above I just received the notifications and then I collected each notification ID, processed them and then I am trying to delete them to free up resources. I am not sure why and how that is failing. Am I using the API wrong? Any tips or ideas?

Also how am I supposed to end to end test the notifier component? The available test mode sends out fake notifcations which don't really exist and I would like to do a full test which I can debug in a reasonable timeframe before shipping this new feature. I have access to a full test environment where I can change the database how I like, so I could also run sql queries against the clair database but I am unsure how best to test this and without breaking the internal state of the database.

Expected Outcome

According to the Clair doc as I understand it, especially https://quay.github.io/clair/reference/api.html, this should work.

Actual Outcome

doesn't work, see above

Environment

• Clair version/image: 4.6.1
• Clair client name/version: keppel
• Host OS: alpine
• Kernel (e.g. uname -a): can't run on the kube node
• Kubernetes version (use kubectl version): 1.25.6
• Network/Firewall setup: it's complicated