Hello, we are running on Azure, in the last few days we get steps that fails with

Error (exit code 1): failed to put file: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

looking at the step logs it finished all the logic and prints the last log line. 1. How can we debug it and understand from where the error is coming from? 2. Could it be that even if we ran in Azure there are some files uploaded to AWS? sine this looks like an S3 error message Thanks

Are you using any custom extension or any custom code that might be trying to communicate to s3?

No that we know of. We are using a custom image with

@kubernetes(image=os.getenv("IMAGE_TAG"))

Init we have the following packages + some internal libraries but none that should connect to S3. This issue is new, we have been running this flow for a few weeks now and started to see it just a few days ago

azure-core==1.26.3
azure-identity==1.12.0
azure-mgmt-storage==20.1.0
azure-storage-blob==12.15.0
azure-common==1.1.28
azure-keyvault-secrets==4.6.0
kubernetes==26.1.0
metaflow==2.8.5
python-dateutil==2.8.2
termcolor==2.2.0
pandas==1.1.5
tenacity==8.2.2
PyGithub==1.55

btw you wouldn't need

metaflow==2.8.5

inside the image

We are running on K8 in Azure with Argo. What do you mean by

extensions outside of the image

? can you give me an example and I will check

The user needs to install metaflow on their laptops. Are they installing any other custom extension alongside?

user install the same packages I listed before

there could be something with the user code potentially? is this happening across all flows or only a few?

Just in a single flow that we are running in high scale. It reads data from 1 DB and copies it to another DB. In every case that I just it seems the error is AFTER the step logic, meaning the last log line from the step logic is in the logs. I tried adding more resources (CPU and memory) and got the same error