Faros does not seem to respect --cascade=false when deleting GitTrack resource

[sebastianrosch]

When running kubectl delete GitTrack <gittrack-name> -n <faros-namespace> --cascade=false, Kubernetes leaves the GitTrackObjects that have an ownerReference to the GitTrack resource untouched. However, if Faros is running, it deletes all GitTrackObjects as well. It seems to ignore the --cascade=false flag.

[JoelSpeed]

Hmm. Faros being on or off shouldn't make a difference to this. Faros doesn't handle any of the deletion of resources, this is all handled (or rather is meant to all be handled) by the Kubernetes Garbage collector.

I think we need to add some proper integration tests to check on this and better diagnose the problem that you are seeing.

Could you please confirm for me the Kubernetes versions you are using for both your API server and Kubectl when seeing this problem? Thanks :)

[aaron-trout]

Hmm yeah I am seeing this behaviour too actually!

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:04:52Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.5-gke.10", GitCommit:"d0686b9f0adfcf759cde9f1d2d80fd52ab01d58f", GitTreeState:"clean", BuildDate:"2019-02-22T20:02:13Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

When I run kubectl delete gittrack <redacted> --cascade=false, I also see the following output from Faros at the same time...

faros-controller-manager-0 manager 2019/04/15 09:38:13 Getting repository '[email protected]:<redacted>'
faros-controller-manager-0 manager 2019/04/15 09:38:13 Checking out 'master'
faros-controller-manager-0 manager 2019/04/15 09:38:14 Updated child for 'sealedsecret-keys'
faros-controller-manager-0 manager 2019/04/15 09:38:14 Updated child for 'namespace-<redacted>'
faros-controller-manager-0 manager 2019/04/15 09:38:14 Updated child for 'deployment-<redacted>'
faros-controller-manager-0 manager 2019/04/15 09:38:14 Updating GitTrack <redacted> status
faros-controller-manager-0 manager 2019/04/15 09:38:14 unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com "<redacted>": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/default/<redacted>, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e6156475-5f60-11e9-b504-42010a9a00e7, UID in object meta:
faros-controller-manager-0 manager 2019/04/15 09:38:14 Updating <redacted>/sealedsecret-keys status
faros-controller-manager-0 manager 2019/04/15 09:38:14 error syncing child <redacted>/sealedsecret-keys: error updating child SealedSecret keys: unable to update child: unable to update child resource: error creating object: the dryRun alpha feature is disabled

[JoelSpeed]

@aaron-trout I haven't managed to look into this issue yet, I'm still confused as to why it could ever happen, Faros doesn't handle deleting those resources, it's all left down to the garbage collector. If there is a bug though, it would be in the cleanup logic in the GitTrack controller which is meant to delete resources no longer in the repo. Needs more investigation!

As you are running on 1.12, you'll need to disable server-dry-run to fix error creating object: the dryRun alpha feature is disabled: add --server-dry-run=false to your args else you won't ever get any updates, you should be able to remove it again from K8s 1.13 😄

[aaron-trout]

@JoelSpeed well, interestingly that also seems to fix the --cascade=false issue 🤔

Now I just get the following from Faros when I delete the GitTrack, but the underlying objects hang around as expected!

unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com "<redacted>": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/default/<redacted>, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 7fc122a9-5f63-11e9-baaa-42010a9a00f9, UID in object meta:

[looztra]

We faced the same issue with --cascade=false not working as expected (and deleting child resources).

╰─» kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.6-gke.13", GitCommit:"fcbc1d20b6bca1936c0317743055ac75aef608ce", GitTreeState:"clean", BuildDate:"2019-06-19T20:50:07Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

faros version v0.4.0-rc5

Associated logs

faros-controller-manager-qa-0 manager E0708 10:57:26.501310       1 gittrack_controller.go:545] gittrack-controller "msg"="error in reconcile" "error"="unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com \"apps-qa\": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/apps-qa/apps-qa, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4206865a-886d-11e9-bff6-42010a030f08, UID in object meta: "
faros-controller-manager-qa-0 manager E0708 10:57:26.501310       1 gittrack_controller.go:545] gittrack-controller "msg"="error in reconcile" "error"="unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com \"apps-qa\": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/apps-qa/apps-qa, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4206865a-886d-11e9-bff6-42010a030f08, UID in object meta: "
faros-controller-manager-qa-0 manager E0708 10:57:26.501310       1 gittrack_controller.go:545] gittrack-controller "msg"="error in reconcile" "error"="unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com \"apps-qa\": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/apps-qa/apps-qa, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4206865a-886d-11e9-bff6-42010a030f08, UID in object meta: "
faros-controller-manager-qa-0 manager E0708 10:57:26.501310       1 gittrack_controller.go:545] gittrack-controller "msg"="error in reconcile" "error"="unable to update GitTrack: Operation cannot be fulfilled on gittracks.faros.pusher.com \"apps-qa\": StorageError: invalid object, Code: 4, Key: /registry/faros.pusher.com/gittracks/apps-qa/apps-qa, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4206865a-886d-11e9-bff6-42010a030f08, UID in object meta: "