Build and push latest version of Docker images on master builds

[whummer]

We need to set up a proper pipeline, to build and push the latest version of the Docker images on green master builds.

[sboagibm]

Edited from the slack discussion:
We need to figure out is tagging and default pull policy. PullAlways will cause learning jobs to deploy slowly all the time. PullIfNotPresent will cause you to use stale images, I think. What we learned in DLaaS is to always use a tagged image, like ":master-36". I had assumed that that's what the vagrant images were doing. Once a image is tagged, it should never change. So you don't have images that are mysteriously changing under you. And if a user reports a problem, you know what code they're running. And then PullIfNotPresent works great. But, yea, that tagging should be part of cd, and I assume is covered by @whummer’s issue.

If you're in a development cycle, deploying to a non-local cluster, the only way to get it there, that I'm aware of, is to push it to a hub. So push in some form needs to be part of the makefiles.

For the modern-tds branch (merge from DLaaS branch) I'm working on, I'm still going back-and-forth in my head what should be the default, maybe PullIfNotPresent should only be the default if minikube is in effect, and otherwise it should be PullAlways? That will make things work for minikube ok, since we'll just replace the existing one when we build, and then PullAlways for a remote cluster, and the dev will use a user-$(WHOAMI) tag for that. And then, when we're doing proper tagging, IMAGE_PULL_POLICY=PullIfNotPresent`` can be the default for production deployment, but remote dev deployment should stay with PullAlways.

I'm also changing, in the makefiles:

IMAGE_TAG ?= latest

to

IMAGE_TAG ?= user-$(WHOAMI)

Which will avoid accidentally pushing a built image to :latest, and will allow the dev to deploy to a remote location without effecting the entire universe.