Maximum number of artifacts exceeded. How to aggregate artifacts from >100 ParallelFor iterations

[bronevet-abc]

/kind bug

I am running a ParallelFor loop with > 100 iterations and aggregating the resulting artifacts into a dsl.Collection but I'm getting the following error:
com.google.cloud.ai.platform.common.errors.AiPlatformException: code=INVALID_ARGUMENT, message=Maximum number of artifacts exceeded. Maximum allowed: 100, requested: 270, cause=null

Is there a way to raise this maximum so I can run larger ParallelFor iterations?

• Kubeflow version: 2.4
• Running on the GCP Vertex Pipelines service

[siddeshas07]

hi @bronevet-abc , I'm not sure but this may help but you can try this you can create a custom container image for your ParallelFor loop execution. This image could handle artifact management internally, bypassing the 100-artifact limit.

[bronevet-abc]

Interesting! I'm not sure exactly the mechanics of this. Are you suggesting a custom container for the task running within the ParallelFor or for the task receiving the aggregated data objects? Can you point to any examples for this or the relevant APIs? Thank you!