-
Notifications
You must be signed in to change notification settings - Fork 918
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failure from hooks are not reflected on UI #1472
Comments
PR's won't be merged. This project is not maintained at this time. Better to form a FORK and DIY.... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
Recently we had a scenario where we did a graceful master takeover, all the processes were okay except the
PostMasterFailoverProcesses
step. The command we had there (which is the last step to update the proxy on our end after takeover) exited with non-zero code. So, on orchestrator UI we saw everything was successful and master was replaced with new one. However since the hook command failed our proxy didn't get updated, the application kept writing to old master and we had to fix it later manually, which is okay for this case on our side.The annoying thing is we didn't see any alerts or notifications on orchestrator UI during the master takeover operation. If we had seen an error or warning there stating the
PostMasterFailoverProcesses
failed we would've taken the action faster.I checked the code a little bit, when we drag a slave in front of the master and trigger a master takeover:
operationResult.responseJSON.Code
from response and ifERROR
it shows an alert:https://github.com/openark/orchestrator/blob/v3.1.4/resources/public/js/orchestrator.js#L171-L173
logic.GracefulMasterTakeover()
and checks any error returned from that func:https://github.com/openark/orchestrator/blob/master/go/http/api.go#L3178
https://github.com/openark/orchestrator/blob/master/go/logic/topology_recovery.go#L2224
executeProcesses()
states that it returns errorin this flow the only problem I see is
logic.GracefulMasterTakeover()
callsexecuteProcesses()
but doesn't check any errors returned. If we have this implemented in the code we would see an alert in the UI when post processes fail and the scenario above can be avoided easily.If you think it makes sense, I will prepare a PR for this.
Orchestrator version: 3.1.4
The text was updated successfully, but these errors were encountered: