Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[External-DNS] Sync mode does not delete/update DNS entries when changes are made on kubernetes side #4430

Open
baptistebalmon opened this issue May 2, 2024 · 3 comments
Open

[External-DNS] Sync mode does not delete/update DNS entries when changes are made on kubernetes side #4430

baptistebalmon opened this issue May 2, 2024 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@baptistebalmon
Copy link

baptistebalmon commented May 2, 2024 •
edited

What happened:
The externalDNS has been configured this way :

image: registry.k8s.io/external-dns/external-dns:v0.14.0
args:
- --policy=sync
- --source=ingress
- --source=service
- --annotation-filter=managed.by=external-dns
- --txt-owner-id=<rke cluster name>-ext-dns
- --txt-prefix=dbaas.

I'm using RFC2136 mode with PowerDNS.

this way, only services and ingresses that have the annotation "managed.by=external-dns" will be managed.
also txt-owner-id allows to identify the RKE Cluster on which the DNS entry should lead to
the txt-prefix allows to create CNAME entries (for ingresses mainly)

The creation works well :

  • 2 TXT records
  • A/CNAME record

Here is an example of a kubernetes service that needs to be created :

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: toto.example.org
    external-dns.alpha.kubernetes.io/ttl: "3600"
    loadbalancer.openstack.org/load-balancer-address: X.X.X.X
    managed.by: external-dns
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  sessionAffinity: None
  type: LoadBalancer

When I delete the kubernetes service or ingress, externalDNS does not delete them on the DNS.
When I update the kubernetes service or ingress, externalDNS does not update them on the DNS but creates a new entry.

What you expected to happen:
On the kubernetes resources (svc/ingress) deletion, cleaning the corresponding A/CNAME and TXT records from the DNS
On the kubernetes resources (svc/ingress) update, updating the corresponding A/CNAME and TXT records from the DNS.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy externalDNS version 0.14.0
  2. Create kubernetes services
  3. Check that the records are created on the DNS
  4. Delete the kubernetes service
  5. Check that the records are not deleted on the DNS

Anything else we need to know?:

Environment:

  • External-DNS version (use external-dns --version): 0.14.0
  • DNS provider: PowerDNS
  • Others: RFC2136
@baptistebalmon baptistebalmon added the kind/bug Categorizes issue or PR as related to a bug. label May 2, 2024
@leonardocaylent
Copy link
Contributor

@baptistebalmon Did you check if this is working correctly with v0.13.6? Can you attach the external-dns pod logs in Debug level?

@baptistebalmon
Copy link
Author

baptistebalmon commented May 7, 2024 •
edited

Hi @leonardocaylent
I've tried to downgrade the externalDNS image version to v0.13.6.
The issue is still the same.
Kubernetes resources are deleted but DNS records are not.
Please find the log in debug mode.

time="2024-05-07T13:44:03Z" level=debug msg="axfr is disabled"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from ingress: toto/haproxy-stats: [toto.org 3600 IN CNAME vip.org []]"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:44:03Z" level=debug msg="ApplyChanges (Create: 9, UpdateOld: 0, UpdateNew: 0, Delete: 0)"
time="2024-05-07T13:44:03Z" level=debug msg="Processing batch 0 of create changes"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto.org 3600 IN CNAME vip.org []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto.org 3600 CNAME vip.org"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.cname-toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.cname-toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:44:03Z" level=debug msg=SendMessage
time="2024-05-07T13:44:03Z" level=debug msg=SendMessage.success

time="2024-05-07T13:49:03Z" level=debug msg="axfr is disabled"
time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:49:03Z" level=debug msg="ApplyChanges (Create: 6, UpdateOld: 0, UpdateNew: 0, Delete: 0)"
time="2024-05-07T13:49:03Z" level=debug msg="Processing batch 0 of create changes"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:49:03Z" level=debug msg=SendMessage
time="2024-05-07T13:49:03Z" level=debug msg=SendMessage.success

Thank you for your help.

@leonardocaylent
Copy link
Contributor

@baptistebalmon I can see that your plan is not detecting anything to delete. I need to ask you 3 more questions:
1)Has this ever worked for you before?
2)Can you test again with the versions v0.12.0 and v0.10.0?
3)If you manually delete the records external-dns creates them again on the next round?

If you can add the creation logs that could also help.

Thank you for reporting this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@baptistebalmon @leonardocaylent