Improved flyte error messages

[cosmicBboy]

Motivation: Why do you think this is important?

Today, error messages are hard to decipher: all the user knows is that something went wrong in a task/workflow. The purpose of this issue is to improve these error messages by:

• providing richer context on where the error occurred exactly
• if applicable, provide a hint about how to address the error

🤝 Community call to action!

If you'd like to contribute to this effort, please create a comment on this issue with the following
information:

0. mention which flyte component needs to be updated (flytekit, flyteadmin, flytepropeller, etc)
1. A minimally reproducible code snippet that produces the error
2. The current error text with full stack trace
3. Ideas on how to improve the error message

Goal: What should the final outcome look like, ideally?

First we need to compile a list of errors to improve!

Describe alternatives you've considered

The alternative is the current state, which makes the UX suffer

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[jboutwell-rva]

Basic pyflyte usage errors can be hard to trace. Described here is using pyflyte with invalid projects or domains.

1. A minimally reproducible code snippet that produces the error
   pyflyte run --remote -p doesnotexist src/examples/basic.py training_workflow --hyperparameters '{"C": 0.1}'
   Code is the example from https://docs.flyte.org/projects/cookbook/en/latest/index.html, but it is reproducible with any task. The error also occurs if you remove the 'flytesnacks' project and run without specifying a project, and the error is the same for an invalid domain.

2. The current error text with full stack trace
   Below is everything printed to the console

Request rejected by the API, due to Invalid input.
	Reason:
	Input Request: {
  "id": {
    "resourceType": "TASK",
    "project": "doesnotexist",
    "domain": "development",
    "name": "basic.get_data",
    "version": "2u4RWAhkSk0LAH4XBzrZOA=="
  },
  "spec": {
    "template": {
      "id": {
        "resourceType": "TASK",
        "project": "doesnotexist",
        "domain": "development",
        "name": "basic.get_data",
        "version": "2u4RWAhkSk0LAH4XBzrZOA=="
      },
      "type": "python-task",
      "metadata": {
        "runtime": {
          "type": "FLYTE_SDK",
          "version": "1.4.2",
          "flavor": "python"
        },
        "retries": {}
      },
      "interface": {
        "inputs": {},
        "outputs": {
          "variables": {
            "o0": {
              "type": {
                "structuredDatasetType": {}
              },
              "description": "o0"
            }
          }
        }
      },
      "container": {
        "image": "cr.flyte.org/flyteorg/flytekit:py3.10-1.4.2",
        "args": [
          "pyflyte-fast-execute",
          "--additional-distribution",
          "s3://dev-flyte-metadata/doesnotexist/development/DTHEXV2MARSVUEH362HBZAQMVQ======/scriptmode.tar.gz",
          "--dest-dir",
          "/root",
          "--",
          "pyflyte-execute",
          "--inputs",
          "{{.input}}",
          "--output-prefix",
          "{{.outputPrefix}}",
          "--raw-output-data-prefix",
          "{{.rawOutputDataPrefix}}",
          "--checkpoint-path",
          "{{.checkpointOutputPrefix}}",
          "--prev-checkpoint",
          "{{.prevCheckpointPrefix}}",
          "--resolver",
          "flytekit.core.python_auto_container.default_task_resolver",
          "--",
          "task-module",
          "basic",
          "task-name",
          "get_data"
        ],
        "resources": {
          "limits": [
            {
              "name": "MEMORY",
              "value": "1Gi"
            }
          ]
        }
      }
    },
    "description": {
      "shortDescription": "Get the wine dataset.",
      "longDescription": {
        "format": "DESCRIPTION_FORMAT_RST"
      }
    }
  }
}

3. Ideas on how to improve the error message
   I would consider specifying an invalid project or domain as simple input errors and should fail quickly with explicit messaging i.e. "Project 'flytesnacks' not found". A full stack trace is hard to mentally parse for such a simple input error even if it did actually say why it failed in the message.

A side request would be some mechanism for changing the default project or domain as 'flytesnacks' doesn't make sense beyond initial setup

[cosmicBboy]

Look into the pyflyte package error when types are not compatible, the error message should have more context on which task/workflow fails

[cosmicBboy]

TODO: move this issue into a discussion to have a persistent place to crowd-source these from the community. Create separate issues for

• Improved flyte error messages #3502 (comment): issue
• type conversion errors: issue
• using promises like python values issue
• using maptasks in a task issue

[zeryx]

Adding some cryptic error messages
Failed with Unknown Exception <class 'TypeError'> Reason: super() argument 1 must be type, not FlyteInvalidInputException
Was detected in 1.4.0, fixed in 1.4.2

[esadler-hbo]

The type transformer errors can be confusing at first. Users need to read between the lines to figure out what is wrong. Let's illustrate this using the following function bad_types.

@task
def bad_types(a: int) -> float:
    return str(a)

Case 1: Output Type Issue

bad_types(a=1)

"""
TypeError: Failed to convert return value for var o0 for function bad_types with error <class 'flytekit.core.type_engine.TypeTransformerFailedError'>: Type of Val '1' is not an instance of <class 'float'>
"""

It is not immediately clear what o0 means.

Case 2: Input Type Ussue

bad_types(a=str(1))

"""
TypeTransformerFailedError: Type of Val '1' is not an instance of <class 'int'>
"""

I don't see the input name a mentioned anywhere. If there was more than 1 input expecting an int, this error could confuse users.

Case 3: Intermediate Values

Structuring the error around the value, e.g. Type of Val '1' requires tracing values through the intermediate tasks of a workflow. For example...

import random

@task 
def add_rand(n: int) -> float:
    return float(n + random.randint(-1000, 1000))

@task
def wf(a: int, b: int):
    bad_types(a=add_rand(n=a), b=b)
    
wf(a=1, b=1)  

"""
TypeTransformerFailedError: Cannot convert literal scalar {
  primitive {
    float_value: -195.0
  }
}
 to <class 'int'>
""" 

I don't know which has the error unless I trace back through the code, which is difficult with complex workflows.

Suggestions

I believe this would be much less confusing if

• It were possible to include the names of inputs variables in the error message to make them immediately clear which variable has a problem.
• Slightly more direction on what to do when they get a type error so new users understand the process of resolving them.

[samhita-alla]

Revise the error message for an empty FlyteDirectory, which currently displays as follows:

Message:

    Failed to get data from s3://my-s3-bucket/data/oe/f288f12b7c3a34060a84-n0-0/38199e55da375a258e98d0cf0281c44b to /tmp/flytee5d8ndoj/local_flytekit/99318925508ad864cd9917aac910f0dc (recursive=True).

Original exception: Access Denied.

User error.

The reason the workflow is failing is because the directory is empty. However, the error message is misleading the user to think that it's due to incorrect permissions.

[samhita-alla]

Run the following code and you should be able to see the error:

import os

import flytekit
from flytekit import task, workflow
from flytekit.types.directory import FlyteDirectory


@task
def t1() -> FlyteDirectory:
    working_dir = flytekit.current_context().working_directory
    dir = os.path.join(working_dir, "flyte_directory")
    os.makedirs(dir, exist_ok=True)
    return FlyteDirectory(path=dir)


@task
def t2(ff: FlyteDirectory):
    ff.download()


@workflow
def wf():
    return t2(ff=t1())

[samhita-alla]

We should enhance the error message for type incompatibility. Here's the code I'm trying to register:

from flytekit import task, workflow

@task
def t1() -> int:
    return "hello"


@task
def t2(x: str) -> str:
    return x


@workflow
def wf():
    t2(x=t1())

I am encountering the following error:

Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
	details: grpc: error while marshaling: proto: Marshal called with nil
	Debug string UNKNOWN:Error received from peer  {grpc_message:"grpc: error while marshaling: proto: Marshal called with nil", grpc_status:13, created_time:"2023-06-16T14:55:36.019174+05:30"}

[ddl-rliu]

I don't mean to revive a very old issue, but I think it is relevant to the following PR to improve audience mismatch debugging: #5078

Old message:

{"json":{"src":"handlers.go:303"},"level":"info","msg":"Failed to parse Access Token from context. 
Will attempt to find IDToken. Error: invalid audience [&{[account] ... 2024-03-22 17:08:22 +0000 
UTC 2024-03-22 16:53:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC ...}]","ts":"2024-03-22T16:53:52Z"}

New message:

{"json":{"src":"handlers.go:303"},"level":"info","msg":"Failed to parse Access Token from context. 
Will attempt to find IDToken. Error: invalid audience [&{[account] ... 2024-03-22 17:08:22 +0000 
UTC 2024-03-22 16:53:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC ...}], wanted ["some-audience"]",
"ts":"2024-03-22T16:53:52Z"}

[kumare3]

Awesome @ddl-rliu