Cluster controller crashes when setting APIServerLoadBalancer.AllowedCIDRs

[capaca]

/kind bug

What steps did you take and what happened:
The latest release so far, 0.9.0 has a bug that needs to be fixed. When setting the OpenStackClusterSpec.APIServerLoadBalancer.AllowedCIDRs property, the load balancer controller tries to read the router IP list to add to add to the list of allowedCIRs from the cluster status but if the router is not set the controller crashes.

I0327 18:50:40.053235       1 controller.go:115] "Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="112c219f-4370-49fa-a0d3-6a6f645f0338/pvasc-teamint-mgl1-02" namespace="112c219f-4370-49fa-a0d3-6a6f645f0338" name="pvasc-teamint-mgl1-02" reconcileID="32c0ab2a-2b73-4689-ae18-5fa9d8087b9b"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1c04fec]

goroutine 249 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 +0x1fa
panic({0x1e1a4e0, 0x35bd790})
	/usr/local/go/src/runtime/panic.go:884 +0x213
sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/loadbalancer.(*Service).getOrUpdateAllowedCIDRS(0xc0007da000, 0xc0002ab400, 0xc0004ce1e0)
	/workspace/pkg/cloud/services/loadbalancer/loadbalancer.go:256 +0xecc
sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/loadbalancer.(*Service).ReconcileLoadBalancer(0xc0007da000, 0xc0002ab400, {0xc000056e00, 0x3a}, 0x192b)
	/workspace/pkg/cloud/services/loadbalancer/loadbalancer.go:143 +0x8c8
sigs.k8s.io/cluster-api-provider-openstack/controllers.reconcileNetworkComponents({0x2466e58, 0xc000521bc0}, 0xc000fa6b60, 0xc0002ab400)
	/workspace/controllers/openstackcluster_controller.go:525 +0x14ce
sigs.k8s.io/cluster-api-provider-openstack/controllers.reconcileNormal({0x2466e58, 0xc000521bc0}, 0x0?, 0xc0002ab400)
	/workspace/controllers/openstackcluster_controller.go:275 +0xf4
sigs.k8s.io/cluster-api-provider-openstack/controllers.(*OpenStackClusterReconciler).Reconcile(0xc000796180, {0x245ddb8, 0xc00065b890}, {{{0xc000d51230?, 0x0?}, {0xc00053bd28?, 0xc0007f1d48?}}})
	/workspace/controllers/openstackcluster_controller.go:127 +0x5e6
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2463430?, {0x245ddb8?, 0xc00065b890?}, {{{0xc000d51230?, 0xb?}, {0xc00053bd28?, 0x0?}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000337f40, {0x245dd10, 0xc000968690}, {0x1ee04e0?, 0xc000be8b80?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316 +0x3ca
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000337f40, {0x245dd10, 0xc000968690})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x587

Related discussion on slack:
https://kubernetes.slack.com/archives/CFKJB65G9/p1710773457623259

What did you expect to happen:
The cluster controller to reconcile the load balancer and set the list of allowed CIDR ranges.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
This issue is already solved on the main branch but we need a fix for the 0.9 release.

Environment:

• Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): v0.9.0
• Cluster-API version: v1.6.0
• OpenStack version: 6.6.0
• Minikube/KIND version: 0.22.0
• Kubernetes version (use kubectl version): v1.29.2
• OS (e.g. from /etc/os-release):