Creating or starting notebook throws 500 error - KeyError: 'message'

[GoIemXIV]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
After notebook creation or clicking launch on existing notebook, an error is thrown and dashboard is empty. Page reloading doesn't help, an error persist some time and blocks any interaction with notebooks.

What did you expect to happen:
After notebook creation or launch existing notebook, a dashboard with an existing notebooks is available and status of the current notebook is changed. Users can interact with UI and manage notebooks in the current namespace

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Stacktrace from jupyter-web-app shows, that there is a problem with container state parsing mechanism:

kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | Caught and unhandled Exception!
kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | 'message'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/src/apps/common/routes/get.py", line 55, in get_notebooks
    contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
  File "/src/apps/common/routes/get.py", line 55, in <listcomp>
    contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
  File "/src/apps/common/utils.py", line 144, in notebook_dict_from_k8s_obj
    "status": status.process_status(notebook),
  File "/src/apps/common/status.py", line 36, in process_status
    status_phase, status_message = get_status_from_container_state(notebook)
  File "/src/apps/common/status.py", line 141, in get_status_from_container_state
    message = container_state["waiting"]["message"]
KeyError: 'message'

Environment:

• Kubeflow version: 1.8.0
• Kubernetes platform: GKE
• Kubernetes version: (use kubectl version): v1.25.10-gke.2700

[thesuperzapper]

@GoIemXIV thanks for raising this, we are tracking a fix for 1.8.1 in #7453 (comment)