Hey folks, I’m having a couple of problems using Metaflow on Azure. I had our DevOps team deploy Metaflow with Kubernetes using the instructions here. They then sent me the end-user setup instructions. Most things seem to work fine. I’m able to use the

@kubernetes

tag and pods spin up on our cluster for steps with the decorator. It also appears that all the flow artifacts are stored in our blob container. Scheduling with Argo also seems to work like a charm. The issues are using the

@conda

decorator and pulling artifacts from the Metaflow API. I’ll give some more detail for these below: 1. Whenever I use the @conda decorator for flows that have steps that run on Kubernetes, it’ll display:

Bootstrapping conda environment...(this could take a few minutes)

for several minutes until I eventually get

Internal error: Azure error: ('Connection aborted.', timeout('The write operation timed out'))

. I can run local flows with

@conda

and flows that have

@kubernetes

steps without issues, but the combination of both seems to cause the issue. 2. When using the Metaflow API, I can see all my flows just fine iterating through the

Metaflow

object, but when I pull out a run for a flow (

HelloFlow

) using

run = Run('HelloFlow/1')

and try

run.data

it’ll return

None

. Additionally, if I try to get the latest successful run from a flow object (

flow

) using

flow.latest_successful_run

, I’ll get the following error:

MetaflowInternalError: Azure error: Operation returned an invalid status 'This request is not authorized to perform this operation using this permission.'
ErrorCode:AuthorizationPermissionMismatch

I think that it’s possible that these are both permissions issues, but wanted to see if you all had some better insight before I pursue that. Thanks in advance!

I will try my best to unblock here.

For 1. the timeout error that happens only when doing

@kubernetes

x

@conda

at the same time. When running in kubernetes, MF will create a conda environment, package everything up and store it in blob storage. Do you have any sense of the size of the conda environment? E.g. perhaps we are uploading gigabytes of conda environment over a slow internet connection? Just one avenue.

For 2. it does look like a permission issue. Is it possible that your client API calls are being made in a config environment that is different from how you run flows? e.g. are you `az login`'d the same way, etc.

Thanks for the help, Jackie! You were right about 2. It seems in the environment I was doing the API calls I didn’t have visibility to the

AZURE_CLIENT_ID

,

AZURE_CLIENT_SECRET

, and

AZURE_TENANT_ID

environmental variables.

As an experiment, what if we use a @conda env that is tiny.... (just pick a small lib, but no numpy).... Just to see if it is a size issue. 312MB is kind of medium. I.e. how long does 300MB get uploaded from your laptop normally?

I tried using just a small library (tinytimer) and got the environment down to 112MB, but still running into the same issue. Runs fine locally and takes maybe 10ish seconds to set up the environment

I think the issue is similar to this. (interaction how large each "write" chunk is vs the timeout for the put operation). We will need to ship a change to tune these numbers a bit (in particular make write timeout period much more forgiving, possibly reducing chunk size as well). Just for interest would you mind sharing your internet upload speed (just for confirmation)

I filed an issue on Github summarizing all details.

I think you’re most likely right here. it seems that my upload speed is 10-15 Megabits per second. By my napkin math, that’s not working within the 30-second timeout for a 100mb+ environment. I’ll try to find faster internet and give it a try. I’ll get back to you with the results. Thanks so much for your help here!

oh great... it's always slow wifi!