[Core feature] Flytekit should support unsafe mode for types

[Mecoli1219]

Tracking issue

Related to flyteorg/flyte#5319

Why are the changes needed?

Use Case: When the user wants to apply Flytekit to their Python codebase, but annotating the type is too hard or time-consuming, we should support unsafe mode for easy usage.

What changes were proposed in this pull request?

1. I add unsafe kwarg to both task and workflow, and the default value of unsafe is False
2. If the unsafe kwarg is True and the types of inputs or outputs are not specified (which is None), replace the type from None to Optional[Any]. This will trigger FlytePickleTransformer to handle each unspecified type.

How was this patch tested?

Two tests are created to examine the ability of unsafe kwarg.

1. test_unsafe_input_wf_and_task: This test mainly examines the unspecified input types. If the unsafe is False, the workflow or task should throw errors. Otherwise, it should succeed.
2. test_unsafe_wf_and_task: This test mimics the use case mentioned above, where the user might not specify all the types in their codebase. The workflow should succeed if the unsafe kwarg of all tasks and workflow are set to True.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

flyteorg/flyte#5319

Docs link

[pingsutw]

Did you test this workflow in the sandbox cluster?

[Mecoli1219]

No, I only test it on my local environment. Can I ask how to try it?

[Future-Outlier]

This PR can help you, you can either use imageSpec or Dockerfile to create your own image.
#1870

[Mecoli1219]

from flytekit import task, Secret, workflow, ImageSpec

flytekit_dev_version = "https://github.com/Mecoli1219/flytekit.git@c7c9a2ad2bdf6799ec324955a023281cca030038"
image = ImageSpec(
    registry="localhost:30000",
    platform="linux/amd64",
    apt_packages=["git"],
    packages=[
        f"git+{flytekit_dev_version}",
    ],
)


@task(unsafe=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0


@workflow(unsafe=True)
def wf1_with_unsafe(a) -> int:
    return t1(a=a)

[Mecoli1219]

The above code will fail. The reason is detailed in flyteorg/flyte#5261.

[Mecoli1219]

from flytekit import task, Secret, workflow, ImageSpec
from typing import Tuple

flytekit_dev_version = "https://github.com/Mecoli1219/flytekit.git@9885189b2af95c0c0dfa94ff92b71e865d95cf80"
image = ImageSpec(
    registry="localhost:30000",
    platform="linux/amd64",
    builder="fast-builder",
    apt_packages=["git"],
    packages=[
        f"git+{flytekit_dev_version}",
    ],
)


@task(unsafe=True, container_image=image)
def t1(a) -> int:
    if type(a) == int:
        return a + 1
    return 0


@workflow
def wf1_with_unsafe() -> Tuple[int, int, int]:
    a1 = t1(a=1)
    a2 = t1(a="1")
    a3 = t1(a=None)    #! This will fail
    return a1, a2, a3

Currently, the input with a value None will fail, but the issue is solved with this PR (flyteorg/flyte#5408).

[Mecoli1219]

@pingsutw I created 2 new unit tests, and both of them passed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.17%. Comparing base (876877a) to head (aa1760c).

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2419       +/-   ##
===========================================
+ Coverage   52.89%   76.17%   +23.28%     
===========================================
  Files         297      182      -115     
  Lines       23957    18574     -5383     
  Branches     3655     3658        +3     
===========================================
+ Hits        12671    14149     +1478     
+ Misses      11178     3796     -7382     
- Partials      108      629      +521     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.