hey guys, I need some help on debugging conda error in metaflow, when I tried to deploy my metaflow pipeline to step functions, I got the below error, so far, I have tried to increase instance duration/timeout from 30 min to 120 min and increase conda timeout by

conda config --set remote_read_timeout_secs 7200.0

, but conda still raises errors, does anyone know what may happen?

could you try by removing the tarball from the target_path and trying again?

@ancient-application-36103 I did not add

target_path

in my metaflow pipeline, not sure how to remove it when metaflow build conda dependency, appreciate any example, thanks

you could just delete it by yourself

it was running on customized GH instance, and it's hard to replicate the env and run the command there

you could try with

conda create -n my_env pytorch

if that command succeeds for you

I have a

conda_base

decorator for entire pipeline and

conda

decorator for each step, should I create a yaml file

environments.yml

with all dependency from these decorator then do

conda env create --name envname --file=environments.yml

?

yep!

@ancient-application-36103 it's interesting that I can deploy my metaflow job from my laptop to AWS step function directly w/o any issues like usual, not sure it raises weird errors on customized github action instance, any idea what may cause the errors?

It's hard to say without knowing the details of the custom gh instance

yup, we have a lot of dependency, some of them are sensitive to system, some are privately hosted (private pip_index_url required), some versions are not available on conda channel etc. Would it possible to use prebuilt image for my metaflow pipeline to skip conda boostrap stage? do you have best practice for complicated dependency management?

yeah you can bring in your image and use

@batch(image='foo')

or

@kubernetes(image='foo')

cool, thanks