Neutron floating IP delete policy cause cluster to stuck in delete progress

[satishdotpatel]

/kind bug

I have a neutron policy which doesn't allow normal users to delete floating IPs.

"delete_floatingip": "(rule:admin_only)"

Now with this policy when I try to delete k8s cluster then it stuck in deleting stats because users not allow to delete floating IPs.

Following are capi logs.

I0402 01:50:03.412447 1 recorder.go:104] "events: Failed to delete floating IP 104.xx.xx.70: Request forbidden: [DELETE https://os2.example.com:9696/v2.0/floatingips/492b2c4c-151b-437e-acfc-4778a45cb9bf], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"rule:delete_floatingip is disallowed by policy\", \"detail\": \"\"}}" type="Warning" object={"kind":"OpenStackMachine","namespace":"magnum-system","name":"kube-6aqje-csgft-nkz6s","uid":"97acabe1-e547-4bbc-8406-aebc563212cd","apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha7","resourceVersion":"31170690"} reason="Faileddeletefloatingip"

What did you expect to happen:

ClusterAPI shouldn't delete floating IP when deleting cluster but it should release floating IP instead.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster API Provider

[EmilienM]

FYI @bilbobrovall

[huxcrux]

/kind bug

I have a neutron policy which doesn't allow normal users to delete floating IPs.

"delete_floatingip": "(rule:admin_only)"

Now with this policy when I try to delete k8s cluster then it stuck in deleting stats because users not allow to delete floating IPs.

Following are capi logs.

I0402 01:50:03.412447 1 recorder.go:104] "events: Failed to delete floating IP 104.xx.xx.70: Request forbidden: [DELETE https://os2.example.com:9696/v2.0/floatingips/492b2c4c-151b-437e-acfc-4778a45cb9bf], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"rule:delete_floatingip is disallowed by policy\", \"detail\": \"\"}}" type="Warning" object={"kind":"OpenStackMachine","namespace":"magnum-system","name":"kube-6aqje-csgft-nkz6s","uid":"97acabe1-e547-4bbc-8406-aebc563212cd","apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha7","resourceVersion":"31170690"} reason="Faileddeletefloatingip"

What did you expect to happen:

ClusterAPI shouldn't delete floating IP when deleting cluster but it should release floating IP instead.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

* Cluster API Provider

Looks like this is v1alpha7 OpenStackMachine meaning this is should not be using the IPAM stuff since it was introduced in v1beta1.

Could you confirm if this is a control-plane machine? and if you have DisableAPIServerFloatingIP set in your OpenstackCluster manifest? (maybe you could share your OpenstackCluster and OpenstackMachineTemplate from the affected cluster(you can OFC truncate the status and remove potential info you do not want to share)?

Also maybe you could confirm what CAPO version you are using?

[mdbooth]

@huxcrux I wonder if this is a use case for the floating IP ipam controller owning the API server floating ip? That's definitely something I wanted to enable when we get around to re-working that stuff.

@satishdotpatel Can you explain why you don't allow deleting FIPs? I'm interested in your use case.

It sounds to me like you want a floating IP with a lifecycle independent of the cluster. If you pre-create a floating IP and set it explicitly in apiServerFloatingIP in the cluster spec the cluster will also not delete it when deleting the cluster.

Incidentally, I note that we haven't documented that behaviour in the docs I linked. Should probably fix that.

[huxcrux]

@huxcrux I wonder if this is a use case for the floating IP ipam controller owning the API server floating ip? That's definitely something I wanted to enable when we get around to re-working that stuff.

@satishdotpatel Can you explain why you don't allow deleting FIPs? I'm interested in your use case.

It sounds to me like you want a floating IP with a lifecycle independent of the cluster. If you pre-create a floating IP and set it explicitly in apiServerFloatingIP in the cluster spec the cluster will also not delete it when deleting the cluster.

Incidentally, I note that we haven't documented that behaviour in the docs I linked. Should probably fix that.

I think that would fix this problem. I assume this is the "old" logic and it seems to always create/delete IPs when the IPAM have a Retain option that simply skips the delete part

[satishdotpatel]

@huxcrux I wonder if this is a use case for the floating IP ipam controller owning the API server floating ip? That's definitely something I wanted to enable when we get around to re-working that stuff.

@satishdotpatel Can you explain why you don't allow deleting FIPs? I'm interested in your use case.

We are running openstack public cloud and because of government regulatory process required accounting of assigned public ips to our customer. When I assigned them list of public IP then I would like them to not delete and re-claim new IP. That is why I put that policy in place to protect delete. I would like them to release floating IP from vm instead delete.

It sounds to me like you want a floating IP with a lifecycle independent of the cluster. If you pre-create a floating IP and set it explicitly in apiServerFloatingIP in the cluster spec the cluster will also not delete it when deleting the cluster.

Incidentally, I note that we haven't documented that behaviour in the docs I linked. Should probably fix that.

Sorry I am new to ClusterAPI so not sure about what you asking. Let me know If you want any info or output of command. I have setup magnum with magnum-cluster-api driver to talk to ClusterAPI k8s management server to provision workload cluster.

I am not sure where we need to fix code to tell not delete floatingIP but release or disassociate them from instance instead of delete them.

[satishdotpatel]

/kind bug
I have a neutron policy which doesn't allow normal users to delete floating IPs.
"delete_floatingip": "(rule:admin_only)"
Now with this policy when I try to delete k8s cluster then it stuck in deleting stats because users not allow to delete floating IPs.
Following are capi logs.
I0402 01:50:03.412447 1 recorder.go:104] "events: Failed to delete floating IP 104.xx.xx.70: Request forbidden: [DELETE https://os2.example.com:9696/v2.0/floatingips/492b2c4c-151b-437e-acfc-4778a45cb9bf], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"rule:delete_floatingip is disallowed by policy\", \"detail\": \"\"}}" type="Warning" object={"kind":"OpenStackMachine","namespace":"magnum-system","name":"kube-6aqje-csgft-nkz6s","uid":"97acabe1-e547-4bbc-8406-aebc563212cd","apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha7","resourceVersion":"31170690"} reason="Faileddeletefloatingip"
What did you expect to happen:
ClusterAPI shouldn't delete floating IP when deleting cluster but it should release floating IP instead.
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:

* Cluster API Provider

Looks like this is v1alpha7 OpenStackMachine meaning this is should not be using the IPAM stuff since it was introduced in v1beta1.

Could you confirm if this is a control-plane machine? and if you have DisableAPIServerFloatingIP set in your OpenstackCluster manifest? (maybe you could share your OpenstackCluster and OpenstackMachineTemplate from the affected cluster(you can OFC truncate the status and remove potential info you do not want to share)?

How do i check DisableAPIServerFloatingIP ? Sorry I am new to ClusterAPI stuff. Give me command and I will get you full output.

Also maybe you could confirm what CAPO version you are using?

capi-openstack/capi-openstack-controller:v0.9.0

[mdbooth]

I actually wrote some documentation around this a while back, but I never followed up and got it pushed. You can find it here, though: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/923a3e2d5b55ad3d19ef3a0b785b429b10ca1c31/docs/book/src/topics/control-plane-endpoint.md

There is a section in there about using a pre-created floating IP, which also mentions that it will not be deleted with the cluster if it is specified in apiServerFloatingIP.

If that documentation is at all helpful to you I'd appreciate your comments on the PR: #1757

I should probably merge it into the main configuration doc, tbh.

[satishdotpatel]

Does pre-create floating IP method works in single master nodes (without octavia LB)? because in that case floating IP will attach to master node vm right?

This is kind of tricky to explain to end user about do pre-create floating IP and attach to master node. Lets say I want to run this test then where do I need to specify floatingIP? I am using magnum-capi driver and there is no way to tell use A floating ip for new cluster.

[mdbooth]

@satishdotpatel The (unpublished) doc I linked above also explains the issue with direct fip attachment right now. We have a plan to fix that, but we haven't started work on it, yet.