Bootstrapping conda environment...(this could take a few minutes)
    Conda ran into an error while setting up environment.:
    Step: write_files, Error: command '['/usr/share/miniconda/condabin/conda', 'create', '--yes', '--no-default-packages', '--name', 'metaflow_ContentRelevancyNNTikTok_linux-64_b03097a5f8dbfe71e4916d813284313ea45ed26a', '--quiet', b'python==3.8.13', b'requests==2.26.0', b'boto3==1.19.12', b'awscli==1.21.12', b'cloudpickle==2.0.0', b'dask==2021.3.0', b'dataclasses-json==0.5.3', b'fastavro==1.4.7', b'fastparquet==0.7.2', b'joblib==1.1.0', b'numpy==1.20.3', b'pandas==1.3.4', b'pip==21.3.1', b'pyparsing==2.4.7', b'pyarrow==6.0.1', b'python-snappy==0.6.0', b'pytz==2021.3', b's3fs==0.4.2', b'scikit-learn==0.23.2', b'scipy==1.7.3', b'snappy==1.1.8', b'smart_open==5.2.1', b'sqlalchemy==1.3.23', b'testpath==0.5.0', b'tornado==6.1', b'tzdata==2021e', b'pycryptodome==3.11.0', b'wandb==0.12.1', b'fastai::fastcore==1.3.26', b'snowflake-connector-python[pandas]==2.7.0', b'snowflake-sqlalchemy==1.2.4', b'great-expectations==0.13.31', b'anaconda::widgetsnbextension==3.5.1', b'jinja2==3.0.3', b'sklearn-pandas==2.2.0', b'pytorch::pytorch==1.9.0']' returned error (-9): b'', stderr=b''

hey guys, one of my metaflow pipelines failed in conda boostrap and returned above error, does anyone know what may cause the errors?

interesting. Can you try nuking the

.metaflow/$flow_name/

folder and trying again to see if somehow the conda envs are in an inconsistent state?

it is from github action. The local build generally works once local conda cache is removed.

Is this an occasional occurrence or is it happening every single time?

it used to work fine, but it fails in almost every build now.

are you able to execute it locally?

yes, I was able to run it locally

when running locally, are you able to run it with

--with batch

(or

kubernetes

). In case you are on a mac, the resolved dependencies would be generated for linux if you do

--with batch

yes, I used

--with batch

in my cmd

One issue that I can think of is that resolving the dependencies might be consuming more memory than before - resulting in the OS killing the conda process. The GH runner instances are often significantly resource underprovisioned.

we have several flows, their dependency is almost the same, but one failed and one succeeded, their GH resources should be same.

even a single version difference for a dependency can result in a very different dependency resolution graph.

is there anyway we can trace back the errors in conda bootstrap?

you can check if

/usr/share/miniconda/condabin/conda create --yes --no-default-packages --name foo python==3.8.13 requests==2.26.0 boto3==1.19.12 awscli==1.21.12 cloudpickle==2.0.0 dask==2021.3.0 dataclasses-json==0.5.3 fastavro==1.4.7 fastparquet==0.7.2 joblib==1.1.0 numpy==1.20.3 pandas==1.3.4 pip==21.3.1 pyparsing==2.4.7 pyarrow==6.0.1 python-snappy==0.6.0 pytz==2021.3 s3fs==0.4.2 scikit-learn==0.23.2 scipy==1.7.3 snappy==1.1.8 smart_open==5.2.1 sqlalchemy==1.3.23 testpath==0.5.0 tornado==6.1 tzdata==2021e pycryptodome==3.11.0 wandb==0.12.1 fastai::fastcore==1.3.26 snowflake-connector-python[pandas]==2.7.0 snowflake-sqlalchemy==1.2.4 great-expectations==0.13.31 anaconda::widgetsnbextension==3.5.1 jinja2==3.0.3 sklearn-pandas==2.2.0 pytorch::pytorch==1.9.0

succeeds in your GH runner

the conda create cmd did work in github actions, all flows succeeded. I am not sure what else caused the conda boostrap errors.

did the flow that was failing, succeed as well?

/usr/share/miniconda/condabin/conda create --yes --no-default-packages --name foo python==3.8.13 requests==2.26.0 boto3==1.19.12 awscli==1.21.12 cloudpickle==2.0.0 dask==2021.3.0 dataclasses-json==0.5.3 fastavro==1.4.7 fastparquet==0.7.2 joblib==1.1.0 numpy==1.20.3 pandas==1.3.4 pip==21.3.1 pyparsing==2.4.7 pyarrow==6.0.1 python-snappy==0.6.0 pytz==2021.3 s3fs==0.4.2 scikit-learn==0.23.2 scipy==1.7.3 snappy==1.1.8 smart_open==5.2.1 sqlalchemy==1.3.23 testpath==0.5.0 tornado==6.1 tzdata==2021e pycryptodome==3.11.0 wandb==0.12.1 fastai::fastcore==1.3.26 snowflake-connector-python[pandas]==2.7.0 snowflake-sqlalchemy==1.2.4 great-expectations==0.13.31 anaconda::widgetsnbextension==3.5.1 jinja2==3.0.3 sklearn-pandas==2.2.0 pytorch::pytorch==1.9.0

this worked fine in GH, but the conda bootstrap still failed, here is the cmd line

how did you execute the conda create command? was it executed as part of a flow that successfully created a conda environment?

I executed conda create by

/usr/share/miniconda/condabin/conda create --yes --no-default-packages --name foo ......

in GH actions, it worked fine. But

python my_script.py --environment=conda package list

raised similar conda bootstrap errors.

One scenario where the previous command may succeed is if any of the packages are already cached on the GH runner instance. But besides that the conda bootstrap function calls the exact same command underneath. It then publishes the packages to s3, but from the stack trace, the error happens during environment creation.

you're right, it turned out to be conda cache issue. I think the previous cache is not compatible with new build even if dependency versions remain the same. It worked fine when conda cache is disabled. Thank you!