https://outerbounds.com/ logo
Join Slack
Powered by
Are there any tips/tricks to resolving conda confl...
# ask-metaflow
w
Are there any tips/tricks to resolving conda conflicts? I'm trying to port my 
pyproject.toml
file into the 
@conda
decorator and it's spewing lots of conflicts
✅ 1
v
are you using 
mamba
? Mamba tends to give nicer error messages than 
conda
w
wasnt using mamba, but i'll give it a go
👍 1
v
it's much faster too
set 
"METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba"
in your Metaflow config
w
should i still use 
CONDA_CHANNELS=anaconda,conda-forge,default
?
a
yep!
d
note that anaconda has license requirements (if that applies to you).
w
mamba doesnt seem any faster than conda. It's been bootstrapping for over 10min now
is there a way to enable verbose mode when bootstrapping the env?
v
what's the error you are seeing?
w
nothing.. it's just stuck for over an hour. I went into the conda.py file to remove the 
--quiet
flag and added some logs. This is what i see:
Copy code
Bootstrapping conda environment...(this could take a few minutes)
/Users/pownissa/mambaforge/bin/mamba ['info']
/Users/pownissa/mambaforge/bin/mamba ['info']
/Users/pownissa/mambaforge/bin/mamba ['info']
/Users/pownissa/mambaforge/bin/mamba ['info']
/Users/pownissa/mambaforge/bin/mamba ['env', 'remove', '--name', 'metaflow_ModelBuilder_osx-64_9d3a3ba046824a40c12d92a065706f31c28664ed', '--yes']
/Users/pownissa/mambaforge/bin/mamba ['create', '--yes', '--no-default-packages', '--name', 'metaflow_ModelBuilder_osx-64_9d3a3ba046824a40c12d92a065706f31c28664ed', b'python==3.10', b'requests==2.27.1', b'boto3==1.24.28', b'click==8.0.3', b'coverage==6.3', b'Jinja2==3.1.2', b'awswrangler==2.17.0', b'bunch==1.0.1', b'cachetools==5.2.0', b'flatten-dict==0.4.2', b'numba==0.56.4', b'numpy==1.23.4', b'omegaconf==2.2.3', b'orjson==3.8.1', b'pydash==5.1.1', b'pytorch-lightning==1.7.7', b'sagemaker-python-sdk==2.77.1', b'scikit-learn==1.1.3', b'scipy==1.9.3', b'tenacity==8.1.0', b'pytorch==1.12.1', b'tqdm==4.64.1']
/Users/pownissa/mambaforge/bin/mamba ['info']
/Users/pownissa/mambaforge/bin/mamba ['list', '--name', 'metaflow_ModelBuilder_osx-64_9d3a3ba046824a40c12d92a065706f31c28664ed', '--explicit']
/Users/pownissa/mambaforge/bin/mamba ['info']
v
btw, do you use Metaflow with s3 or locally? Are you running this on a cloud instance or a laptop? it'll upload packages to S3, which can also take a while over a slow network connection
an hour sounds extreme though
I took your list of packages and ran
Copy code
mamba create -n testenv python==3.10 requests==2.27.1 boto3==1.24.28 click==8.0.3 coverage==6.3 Jinja2==3.1.2 awswrangler==2.17.0 bunch==1.0.1 cachetools==5.2.0 flatten-dict==0.4.2 numba==0.56.4 numpy==1.23.4 omegaconf==2.2.3 orjson==3.8.1 pydash==5.1.1 pytorch-lightning==1.7.7 sagemaker-python-sdk==2.77.1 scikit-learn==1.1.3 scipy==1.9.3 tenacity==8.1.0 pytorch==1.12.1 tqdm==4.64.1
and it took a few minutes to create, so it doesn't seem that resolving the dependencies is the issue
w
I'm just running it on my laptop. Just checked, my upload speed is 160Mbps. Is there any way to see what it's actually stuck on?
v
you can try the above command manually too
w
i tried that mamba create command, and it works
forgot to mention, i'm just trying to run my DAG locally:
Copy code
METAFLOW_CONDA_DEPENDENCY_RESOLVER=mamba CONDA_CHANNELS=anaconda,conda-forge,default TEST_DATA_PATH=small.parquet python model_builder.py --environment=conda run --base_cfg ./conf/models/Lightsabre/Base.yaml --override_cfg ./conf/test.yaml
v
I'm trying to reproduce the issue
w
here's my simplified DAG:
Copy code
from metaflow import (
    FlowSpec,
    step,
    batch,
    retry,
    schedule,
    timeout,
    project,
    conda_base,
    Parameter,
    JSONType,
)
from datetime import datetime, timedelta

import json


@schedule(hourly=True)
@project(name="model_builder")
@conda_base(
    libraries={
        "boto3": "1.24.28",
        "click": "8.0.3",
        "coverage": "6.3",
        "requests": "2.27.1",
        "Jinja2": "3.1.2",
        "awswrangler": "2.17.0",
        "bunch": "1.0.1",
        "cachetools": "5.2.0",
        "flatten-dict": "0.4.2",
        "numba": "0.56.4",
        "numpy": "1.23.4",
        "omegaconf": "2.2.3",
        "orjson": "3.8.1",
        "pydash": "5.1.1",
        "pytorch-lightning": "1.7.7",
        "sagemaker-python-sdk": "2.77.1",
        "scikit-learn": "1.1.3",
        "scipy": "1.9.3",
        "tenacity": "8.1.0",
        "pytorch": "1.12.1",
        "tqdm": "4.64.1",
    },
    python="3.10",
)
class ModelBuilder(FlowSpec):
    base_cfg = Parameter("base_cfg", default="./conf/models/Lightsabre/Base.yaml")
    override_cfg = Parameter("override_cfg", default="./conf/test.yaml")

    @step
    def start(self):
        self.run_date = datetime.utcnow().replace(microsecond=0, second=0, minute=0)
        self.next(self.preprocess_data)

    @timeout(minutes=180)
    @batch(cpu=96, memory=256_000)
    @step
    def preprocess_data(self):
        self.next(self.train_model)

    @timeout(minutes=60)
    @batch(gpu=1)
    @step
    def train_model(self):
        self.next(self.deploy_model)

    @retry(times=4)
    @step
    def deploy_model(self):
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    ModelBuilder()
Copy code
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:54) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import metaflow
metaf>>> metaflow.__version__
'2.7.15'
v
it seems to be impossibly slow with 
conda
4.9.2 but seems to work fine with 
mamba
1.0.0
what's your version of 
mamba
?
w
Copy code
mamba --version
mamba 1.0.0
conda 22.9.0
Copy code
active environment : base
    active env location : /Users/pownissa/mambaforge
            shell level : 1
       user config file : /Users/pownissa/.condarc
 populated config files : /Users/pownissa/mambaforge/.condarc
                          /Users/pownissa/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.10.6.final.0
       virtual packages : __osx=13.0=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /Users/pownissa/mambaforge  (writable)
      conda av data dir : /Users/pownissa/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : <https://conda.anaconda.org/conda-forge/osx-64>
                          <https://conda.anaconda.org/conda-forge/noarch>
                          <https://repo.anaconda.com/pkgs/main/osx-64>
                          <https://repo.anaconda.com/pkgs/main/noarch>
                          <https://repo.anaconda.com/pkgs/r/osx-64>
                          <https://repo.anaconda.com/pkgs/r/noarch>
          package cache : /Users/pownissa/mambaforge/pkgs
                          /Users/pownissa/.conda/pkgs
       envs directories : /Users/pownissa/mambaforge/envs
                          /Users/pownissa/.conda/envs
               platform : osx-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.10.6 Darwin/22.1.0 OSX/13.0
                UID:GID : 93773924:1896053708
             netrc file : None
           offline mode : False
v
I wonder if the S3 uploading is the slow part. Try this:
Copy code
METAFLOW_CONDA_DEPENDENCY_RESOLVER=mamba CONDA_CHANNELS=anaconda,conda-forge,defaults python flow.py --environment=conda --datastore=local --metadata=local run
w
Copy code
AWS Batch error:
    The @batch decorator requires --datastore=s3.
v
comment out 
@batch
to test it
✅ 1
w
well, that worked 🤦‍♂️
v
ok, great! Can you test a basic flow that doesn't use 
@conda
to make sure your AWS/S3 config works otherwise?
w
i tried from EC2 and the flow works (with the 
@batch
decorator), so guess it's just my internet
v
yeah, it tries to upload a big bunch of stuff in parallel which may overwhelm the local network
w
good that we figured out the problem, but rather annoying that i can't test locally (unless i comment out the batch decorator)
another thing i noticed is that the packages available on linux are different to osx. One of my dependences (
numba 0.56.4
) wasn't satisfying. How do people work around this?
Copy code
Step: start, Error: command '['/local/home/pownissa/mambaforge/bin/mamba', 'create', '--yes', '--no-default-packages', '--name', 'metaflow_ModelBuilder_linux-64_9d3a3ba046824a40c12d92a065706f31c28664ed', '--quiet', b'python==3.10', b'requests==2.27.1', b'boto3==1.24.28', b'click==8.0.3', b'coverage==6.3', b'Jinja2==3.1.2', b'awswrangler==2.17.0', b'bunch==1.0.1', b'cachetools==5.2.0', b'flatten-dict==0.4.2', b'numba==0.56.4', b'numpy==1.23.4', b'omegaconf==2.2.3', b'orjson==3.8.1', b'pydash==5.1.1', b'pytorch-lightning==1.7.7', b'sagemaker-python-sdk==2.77.1', b'scikit-learn==1.1.3', b'scipy==1.9.3', b'tenacity==8.1.0', b'pytorch==1.12.1', b'tqdm==4.64.1']' returned error (1): b'Encountered problems while solving:\n  - nothing provides requested numba 0.56.4\n\n{\n    "success": false\n}\n', stderr=b''
v
yeah, there's an inevitable discrepancy when running stuff across os x and cloud instances that use linux there are a few solutions that help with that issue, as well as enable 
@batch
locally in your case: 1. you could bake in the dependencies in a Docker image and use 
@batch(image=...)
2. you could use a local editor with EC2 as a remote backend 3. use a cloud workstation (Google colab / Sagemaker / Metaflow sandbox etc)
w
if i go with the Docker image approach, how does it know what version of python to use? is it just the deafult 
python
? so i have to install all the deps globally?
v
yep. You can use Conda/pip/poetry whatever in your docker as long as the PATH points at 
python
in that environment
in general there's no much need to use a virtual environment inside a Docker image unless you want to use one image for many different use cases
w
It would be nice if we didn't have to create the image in advance. Like the batch step could just install everything from my poetry file as the very first thing it does.
v
that's pretty much what 
@conda
does, except that it resolves the required packages only once which makes things much faster (imagine having a foreach with many tasks, all of which have to resolve all dependencies from scratch) the issue here is that the environment is large enough that uploading from your local laptop seems to cause hiccups 😢
you can do 
subprocess.call(“pip install…”)
n your step code as a workaround
just make sure you do it before importing any libraries you are installing
w
maybe we can make the resolve/upload step happen in the cloud? it only needs whatever is in the 
@conda
dict for that right?
v
yep, that’s an option but it has a bunch of complexity of its own: imagine 
start
step running locally but foreach running on 
@batch
- where would the resolution happen in that case
@wooden-dusk-90720 did you the 
subprocess.call("pip install")
approach? It might be the easiest solution in your case?
w
i'm just going to work off EC2, so basically option (2) you listed
v
cool, that makes life easier 👍
let us know if you hit any further hiccups
👍 1
2 Views
Open in Slack
PreviousNext
Powered by