Hello team. I am trying to optimise metaflow infrastructure for running tasks at very high volume (let’s say each step takes only <10s and there is a large fanout of steps). Let’s disregard whether this is a right thing to do from an application architecture wise, the bottleneck I am current facing is that at the start of step, metaflow seems to run these commands (for context I am on AWS SFN + Batch over EC2)

mflog 'Setting up task environment.' && python -m pip install awscli requests boto3 -qqq && ...

If my EC2 machine is overloaded with more than 30-40 tasks this step starts taking around 1 minute due to congested network. As you can imagine I am reluctant to pay an overhead of 1 minute when my total task runtime is <10s. Do you guys have any ideas on how I can avoid this install? We package all our application-level dependencies via a docker container, I am willing to package these dependnecies into it as well if there’s a way to disable this statement.

💯 that's our recommendation as well. the task tries to install the missing dependencies at bootup - to short circuit it, just make sure all the dependencies are present inside the image.

@quick-lighter-52296 @square-wire-39606 I tried to do the same but not sure why it's not picking it up. I have awscli, boto3 and requests part of the docker image. If I manually modify the Argo template and delete that line which does the

'Setting up task environment.' && python -m pip install awscli requests boto3 -qqq && ...

it works well and I can see a clear improvement in the duration of the step, however if I leave that pip install in there it doesn't seem to recognize that it already has these libs installed. Do I need to add the path to the libs or something like that? Looking at logs the

Settings up task...

takes 14s as opposed to a few ms if I remove that pip install.