https://outerbounds.com/ logo
Join Slack
Powered by
Hi team, Some of our demo flows that were once wo...
# ask-metaflow
a
Hi team, Some of our demo flows that were once working no longer work due to some conda external dependency issues. I'm trying to resolve these issues. Have a couple inquiries: • How to install 
pytorch-cuda
with just the 
@conda
decorator ◦ Keep getting an error relating to not being able to install 
cudart
• I tried using the metaflow bleeding edge extensions and tried resolving the environment yml file i.e. (
CONDA_OVERRIDE_CUDA=11.8 metaflow environment resolve --dry-run -f env.yml --python 3.10.1
) ◦ But I get this error:
Copy code
raise CondaException(
metaflow_extensions.netflix_ext.plugins.conda.utils.CondaException: Cannot resolve environments in mixed mode because no resolver is configured
f
that error is complaining about 
CONDA_MIXED_DEPENDENCY_RESOLVER
not being set, can search for that in the repo readme - if your env.yml has both conda and pip dependencies you need to set that env variable to 
conda-lock
(and have it installed in the env you're running metaflow in)
a
Ah right that makes sense, forgot about that - thanks Bryan!
🎉 1
d
thanks @flaky-plumber-70709, yes, you need to set it to 
conda-lock
. I can set it by default maybe. Initially I had disabled it because the mainline didn’t want it but now that it is an extension, it can just be enabled by default 🙂
let me know if it doesn’t work.
a
Having problems with this env.yml:
Copy code
name: t5
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - pip
  - metaflow
  - pandas=1.5.3
  - matplotlib=3.5.3
  - pytorch-cuda=11.8
  - torchvision=0.14.1
  - pytorch-lightning=1.9.4
  - sentencepiece=0.1.97
  - scikit-learn=1.2.2
Copy code
Resolving 1 environment ...Pretty-printed STDOUT:
Could not solve for environment specs
The following packages are incompatible
└─ pytorch-cuda 11.8  is uninstallable because there are no viable options
   ├─ pytorch-cuda 11.8 would require
   │  └─ cuda-cudart >=11.8,<12.0 , which does not exist (perhaps a missing channel);
   └─ pytorch-cuda 11.8 would require
      └─ cuda 11.8.* , which does not exist (perhaps a missing channel).
{
    "success": false
}
I tried installing 
cuda-cudart
, but still errors out
f
how did you install cuda-cudart?
a
`conda-forge::cuda-cudart`: 
11.8
d
let me try it here locally. one sec. As a quick note, you should probably not include metaflow in your dep file.
👍 1
are you running this on a GPU machine?
a
No, Sagemaker Studio custom kernel. So it's running on a linux based docker
d
I think you are missing the nvidia channel
let me check
a
This used to work 
nvidia::pytorch-cuda
a few months ago, but now mamba can't pick this up, so I removed it.
d
this works:
Copy code
iname: t5
channels:
  - pytorch
  - conda-forge
  - nvidia
dependencies:
  - pandas=1.5.3
  - matplotlib=3.5.3
  - pytorch-cuda=11.8
  - torchvision=0.14.1
  - pytorch-lightning=1.9.4
  - sentencepiece=0.1.97
  - scikit-learn=1.2.2
conda-forge does not have a cuda-cudart package that is <12
a
Yup, thanks Romain, works for me too. Going to try running the Metaflow flow with this now
d
cool — let me know if you face more issues.
thankyou 1
🙏 1
f
if you're running a custom kernel I think you may need to have cuda support baked into the image (when running the flow locally on sagemaker)
a
I think you might be right Bryan - I'll look into that.
So running the actual flow, it cannot find 
pytorch-cuda
:
Copy code
Metaflow 2.10.3+netflix-ext(1.0.5) executing T5FSDPFlow for user:hunr592
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ...Pretty-printed STDOUT:
Could not solve for environment specs
The following package could not be installed
└─ pytorch-cuda 11.8  does not exist (perhaps a typo or a missing channel).
{
    "success": false
}
This is the 
conda_base
:
Copy code
@conda_base(
    libraries={
        "nvidia::pytorch-cuda": "11.8",
        "pytorch::torchvision": "0.14.1",
        "conda-forge::matplotlib": "3.5.3",
        "conda-forge::sentencepiece": "0.1.97",
        "conda-forge::pandas": "1.5.3",
        "conda-forge::pytorch-lightning": "1.9.4",
        "conda-forge::scikit-learn": "1.2.2"
    },
    python="3.10.1",
)
f
maybe they yanked it? don't see it listed in the nvidia channel: https://anaconda.org/nvidia/repo?page=1
see 11.8 at pytorch::pytorch-cuda though: https://anaconda.org/pytorch/pytorch-cuda/files?version=11.8
but you had that already in your env file
a
Yup that's right.
f
when there's a full on whodunit mystery with conda resolvers + metaflow I switch out from mamba as a last roll of the 🎲
a
Tried switching back to 
conda
resolver, but that leads to another set of errors. 😭
f
oi
don't know why this might work exactly but maybe it's related to your kernel not having gpu/cuda support so as a hail mary can try using one of sagemakers dl containers w that included: https://github.com/aws/deep-learning-containers/blob/master/available_images.md
theres some pytorch+gpu labeled kernels IIRC
a
Yeah, thanks, It's probably related to that. 😕
d
sorry for the delay
You can do it this way:
Copy code
@conda_base(
    libraries={
        "pytorch::pytorch-cuda": "11.8",
        "pytorch::torchvision": "0.14.1",
        "conda-forge::matplotlib": "3.5.3",
        "conda-forge::sentencepiece": "0.1.97",
        "conda-forge::pandas": "1.5.3",
        "conda-forge::pytorch-lightning": "1.9.4",
        "conda-forge::scikit-learn": "1.2.2"},
    channels=["nvidia"],
    python="3.10.1",
)
or to have exactly what you had:
Copy code
@conda_base(
    libraries={
        "pytorch-cuda": "11.8",
        "torchvision": "0.14.1",
        "matplotlib": "3.5.3",
        "sentencepiece": "0.1.97",
        "pandas": "1.5.3",
        "pytorch-lightning": "1.9.4",
        "scikit-learn": "1.2.2"},
    channels=["nvidia", "pytorch"],
    python="3.10.1",
)
For some reason (not sure why but probably internals of mamba), the second version seems to solve faster. The issue btw is that the nvidia channel is needed for the cuda-cudart package so you need to list that channel somewhere. As @flaky-plumber-70709 noted, the pytorch-cuda is no longer in the nvidia channel for some reason. Anyways, so I tested and this does work.
❤️ 1
a
Thank you so much Romain
What does this error mean?
Copy code
2023-10-20 01:58:32.630 Workflow starting (run-id 23), see it in the UI at <https://d-khdgmtiq3x4i.studio.us-east-1.sagemaker.aws/jupyter/default/metaflow/T5FSDPFlow/23>
2023-10-20 01:58:32.929 Creating Conda environment ec19849b037e808f11118323f1b5be5c4762a101 (8082e5a9406467981ffcc4b61d7cb0659290b6cb)...
WARNING: conda/mamba do not properly handle installing .conda packages in offline mode. Creating environments may fail -- if so, please install `micromamba`. See <https://github.com/conda/conda/issues/11775>.
    Extracting and linking Conda environment ...No STDOUT
Pretty-printed STDERR:
EnforceUnusedAdapter called with url <https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h807b86a_2.conda>
This command is using a remote connection in offline mode.
d
It is a bug in conda and mamba. I reported it and it is a one line fix but then forgot about it and I needed to add tests and sign an agreement or something. Anyways, I’ll pull it up later and maybe I should actually fix it for them. The solution though is to have micromamba as well which doesn’t suffer from the bug. Mamba is better to resolve imho but micromamba is better to hydrate/install so if you have both it should work as expected.
a
Ok will have to add micromamba to the kernel then. Right now I only have mamba to work with.
In any case, I did manage to get the virtual env resolved just by adding 
conda config --env --add channels nvidia
There are some new issues that popped up though: • 
torch.cuda.is_available()
is returning False (even though I seemingly installed 
pytorch-cuda
). • And it can't seem to find 
torchrun
even though I installed 
pytorch
f
do you know if the image you’re using is set up for cuda?
can try that flow on one of the sagemaker PyTorch gpu kernels now that you have the conda bits figured out
👌 1
a
Usually (in the past), even if the batch image is not set up for cuda, I can get cuda working with the below:
Copy code
@pip(
        libraries={
            "opencv_python_headless": "4.5.5.62"
        }
    )
    @environment(
        vars={
            "EN_BATCH": os.getenv("EN_BATCH"),
            "NVIDIA_DRIVER_CAPABILITIES": "compute,utility",
            "CUDA_HOME": "/usr/local/cuda"
        }
    )
I'm basically just trying to get previously working flows back up and running again. I'm going to first try lowering the version of 
pytorch-cuda
to 11.6. Might be version incompatibility
d
For micromamba, it’s just a binary so should be easy to get in. I can also add an always install. I was hoping they fixed the bug :).
a
Ok good news - my intuition was right. I downgraded 
pytorch-cuda
to 11.6 and now cuda is available. The only remaining error is this:
Copy code
No such file or directory: 'torchrun'
So I think there's a bug where if you use 
subprocess.run
, it can't import the external dependencies from the virtual environment. Tried downgrading to an older version of Metaflow 
2.9.10
, but get this error:
Copy code
2023-10-20 04:43:19.017 [34/train/control-34-start-136 (pid 12480)] [0abdd5d8-bc98-40a5-897f-7b86bf7ab854] Code package downloaded.
2023-10-20 04:43:19.073 [34/train/control-34-start-136 (pid 12480)] [0abdd5d8-bc98-40a5-897f-7b86bf7ab854] Task is starting.
2023-10-20 04:43:19.705 [34/train/control-34-start-136 (pid 12480)] [0abdd5d8-bc98-40a5-897f-7b86bf7ab854] Bootstrapping virtual environment...
2023-10-20 04:43:19.705 [34/train/control-34-start-136 (pid 12480)] [0abdd5d8-bc98-40a5-897f-7b86bf7ab854] /usr/local/bin/python: Error while finding module specification for 'metaflow.plugins.pypi.bootstrap' (ModuleNotFoundError: No module named 'metaflow.plugins.pypi')
d
For torchrun, are you saying you do something like subprocess to launch this torchrun binary which should be in the bin/ directory in the conda env?
a
Yup exactly. Used to work fine, but doesn't seem to work now.
d
Is this with the normal decorators, the extension one or both? I wonder if path isn’t being set but I don’t recall it being set before so it’s bizarre. This is also all running locally correct?
a
Normal decorator and running on 
@batch
It's basically this flow - https://github.com/rileyhun/llm_finetuning_metaflow/blob/722306041c4f00eb9bf791445fed5d6ac81d28c9/pytorch-ddp/ddp_flow.py#L155 I can usually use 
subprocess
to launch the distributed training job, but unfortunately, it cannot find torch or torchrun or any of the dependencies I specified in the 
@conda
decorator
d
ok, so I checked and it’s possible there is a bug in the normal decorators (see here: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/conda_environment.py#L326). I don’t see it being set previously so I am not entirely sure how it ran then. It should be a simple fix. The extension one should work though. I tried this simple stupid flow:
Copy code
import os
import subprocess

from metaflow import FlowSpec, conda, step

class TestSubprocess(FlowSpec):

    @conda(libraries={"tree": ""})
    @step
    def start(self):
        print("Got PATH as %s" % os.environ.get("PATH", "WTC"))
        subprocess.run(["tree", "-L", "1", "/"])
        self.next(self.end)

    @step
    def end(self):
        print("All done")

if __name__ == "__main__":
    TestSubprocess()
and when running 
python ./riley.py --environment=conda run --with batch
, I get (relevant parts only):
Copy code
2023-10-19 22:47:19.006 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] Environment bootstrapped.
2023-10-19 22:47:19.765 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] Got PATH as /root/micromamba/envs/metaflow_27bb27f9eeb1ec64a594a407adac09000a91214b_2529750338903c0626c8826c0d9ac72051344083/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-10-19 22:47:19.767 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] /
2023-10-19 22:47:19.767 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── bin -> usr/bin
2023-10-19 22:47:19.767 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── boot
2023-10-19 22:47:19.767 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── dev
2023-10-19 22:47:19.767 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── etc
2023-10-19 22:47:19.768 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── home
2023-10-19 22:47:19.768 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── lib -> usr/lib
2023-10-19 22:47:19.768 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── lib32 -> usr/lib32
2023-10-19 22:47:19.768 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── lib64 -> usr/lib64
2023-10-19 22:47:19.768 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── libx32 -> usr/libx32
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── media
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── metaflow
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── mnt
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── opt
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── proc
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── root
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── run
2023-10-19 22:47:19.769 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── sbin -> usr/sbin
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── srv
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── sys
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── tmp
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] ├── usr
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] └── var
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352]
2023-10-19 22:47:19.770 [280/start/5173 (pid 63715)] [6b1c62d7-5ff0-4ae1-96c3-64f564c2c352] 23 directories, 0 files
and you can see that the PATH value seems to be set correctly.
(your flow seemed a little more complicated to try 🙂 )
a
Ok thanks Romain - I'll add micromamba to the image. so that I can use the experimental decorator. This is very helpful!
d
let me see if I can also just install micromamba if it is not there. It’s stupid. This is the issue btw: https://github.com/conda/conda/issues/11775
🙏 1
it’s been a year. Geeessh. I got really distracted.
a
Most appreciated!
d
gimme a few minutes. testing it out.
🙏 1
a
How did you add channels to the 
conda_base
decorator? I have the experimental extension installed, but getting this error:
Copy code
metaflow.exception.InvalidDecoratorAttribute: Decorator 'conda_base' does not support the attribute 'channels'. These attributes are supported: packages, libraries, python, disabled.
d
are you sure the experimental ones are installed? That message comes from the mainline decorators
does the version string list the experimental version ? (netflix_ext(1.0.5))?
ok, I’ve finished testing this small change to install micromamba. I’ll be pushing that out.
a
Yup -
Copy code
Warning: 'nvidia' already in 'channels' list, moving to the top
Metaflow 2.10.3+netflix-ext(1.0.5) executing T5FSDPFlow for user:hunr
Now I'm getting this error -
Copy code
2023-10-20 06:27:11.350 [39/train/control-39-start-154 (pid 2812)] Unknown step decorator:
2023-10-20 06:27:11.350 [39/train/control-39-start-154 (pid 2812)] Unknown step decorator conda_env_internal. The following decorators are supported: pytorch_parallel, pypi, metaflow_ray, secrets, environment, parallel, catch, retry, resources, timeout, conda, batch, card, kubernetes
Maybe I should run 
rm -rf .metaflow
d
I think something got screwed up in your installation. You can remove the 
.metaflow
if you want but I would also do an uninstall of metaflow and the extension and reinstall both. I can further debug if you want (you can send me the log of 
METAFLOW_DEBUG_EXT=1 …
but reinstall should fix it.
a
Okay sounds good to me. I'll reinstall both
d
(and any other if you have another extension)
👌 1
1.0.6 should be out shortly which fixes the micromamba issue you brought up.
thankyou 1
sorry for all the back and forth, it shouldn’t be this hard 😞
a
Now I'm getting this error after re-installing and then installing:
Copy code
Traceback (most recent call last):
  File "/home/sagemaker-user/metaflow_demos/src/fsdp-multi-node-multi-gpu/fsdp_flow.py", line 4, in <module>
    from metaflow import (
  File "/usr/local/lib/python3.11/site-packages/metaflow/__init__.py", line 54, in <module>
    from metaflow.extension_support import (
  File "/usr/local/lib/python3.11/site-packages/metaflow/extension_support/__init__.py", line 799, in <module>
    _all_packages, _pkgs_per_extension_point = _get_extension_packages()
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/extension_support/__init__.py", line 790, in _get_extension_packages
    raise RuntimeError(
RuntimeError: Package at '/home/sagemaker-user/.local/lib/python3.11/site-packages/metaflow_extensions' does not define a configuration file for 'config'
And no worries! I really appreciate your help on this
d
what’s in that directory?
the message indicates that it’s finding some extension in metaflow_extensions that is not totally legit.
a
Screenshot 2023-10-20 at 12.23.29 AM.png
d
could you rerun with 
METAFLOW_DEBUG_EXT=1 python -c 'import metaflow'
?
a
Ok I rebooted the studio image and don't get that error anymore.
What does this mean?
d
ah, ok, that one is more classic 🙂
you can stop your current process and then locate that lock file and remove it.
a
Ok sounds good
d
it’s at that path. It just means that some other process may have left it behind.
there are a few lock files. Hopefully that’s the only one left behind. Otherwise it will tell you for all the ones that are there.
👌 1
it’s because we support running multiple flows in parallel and that requires some synchronization. My internal tests solve thirty odd environments in parallel to stress test it (in fairness, there is still some issue somewhere that I haven’t found but it’s rare enough 🙂 )
a
Ah ok, makes sense. Thanks for the explanation. 🙂
Got this error:
Copy code
Metaflow 2.10.2+netflix-ext(1.0.6) executing T5FSDPFlow for user:hunr592
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    All packages already cached in s3.
    All environments already cached in s3.
Including file data/test.csv of size 3MB 
Including file data/train.csv of size 25MB 
Including file data/val.csv of size 3MB 
2023-10-20 07:41:20.694 Workflow starting (run-id 45), see it in the UI at <https://d-khdgmtiq3x4i.studio.us-east-1.sagemaker.aws/jupyter/default/metaflow/T5FSDPFlow/45>
2023-10-20 07:41:20.937 Creating Conda environment 164cd1aff6018b43f3b03f0eff957621a88fe583 (2298fbd3713492c0d15cf783a145284dd622adb3)...
    Downloading 0(web) + 208(cache) packages for arch linux-64 ... done in 228 seconds.
WARNING: conda/mamba do not properly handle installing .conda packages in offline mode (See <https://github.com/conda/conda/issues/11775>).
2023-10-20 07:45:12.977 Workflow failed.
2023-10-20 07:45:12.977 Terminating 0 active tasks...
2023-10-20 07:45:12.977 Flushing logs...
    Internal error
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/cli.py", line 850, in run
    runtime.execute()
  File "/usr/local/lib/python3.11/site-packages/metaflow/runtime.py", line 240, in execute
    self._launch_workers()
  File "/usr/local/lib/python3.11/site-packages/metaflow/runtime.py", line 646, in _launch_workers
    self._launch_worker(task)
  File "/usr/local/lib/python3.11/site-packages/metaflow/runtime.py", line 672, in _launch_worker
    worker = Worker(task, self._max_log_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/runtime.py", line 1183, in __init__
    self._proc = self._launch()
                 ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/metaflow/runtime.py", line 1230, in _launch
    deco.runtime_step_cli(
  File "/home/sagemaker-user/.local/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py", line 525, in runtime_step_cli
    conda.create_for_step(self._step_name, resolved_env),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sagemaker-user/.local/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 380, in create_for_step
    return self.create_for_name(env_name, env, do_symlink)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sagemaker-user/.local/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 421, in create_for_name
    env_path = self._create(env, name)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sagemaker-user/.local/lib/python3.11/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2336, in _create
    % self._bins["micromamba"]
      ~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'micromamba'
taking a look 1
d
I am a real idiot. Give me a minute. I test, it works and I change something thinking: “oh this will be better”. Moron.
a
Not at all. Super obliged just to be getting your help and troubleshooting expertise.
d
1.0.7 should be out in <1min which fixes this
🙏 1
a
Have you seen this error before?
Copy code
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/pkgs/nsight-compute-2023.3.0.12-0/info/repodata_record.json'
d
No. When does it show up?
a
Okay good news the micromamba part works. But still errors out when it hits the batch step:
Copy code
Metaflow 2.10.3+netflix-ext(1.0.7) executing T5FSDPFlow for user:hunr592
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ... done in 23 seconds.
    Downloading 159(web) + 0(cache) packages for arch linux-64 ... done in 31 seconds.
    Caching 165 items to s3 ... done in 133 seconds.
    Caching 2 environments and aliases to s3 ... done in 0 seconds.
Including file data/test.csv of size 3MB 
Including file data/train.csv of size 25MB 
Including file data/val.csv of size 3MB 
2023-10-20 08:18:06.815 Workflow starting (run-id 1266), see it in the UI at <https://d-xxikydhjgmkz.studio.us-west-2.sagemaker.aws/jupyter/default/metaflow/T5FSDPFlow/1266>
2023-10-20 08:18:07.033 Creating Conda environment 164cd1aff6018b43f3b03f0eff957621a88fe583 (34625e4e745f6f968db3b9dddfdbd36e754f9112)...
    Downloading 0(web) + 49(cache) packages for arch linux-64 ... done in 7 seconds.
WARNING: conda/mamba do not properly handle installing .conda packages in offline mode (See <https://github.com/conda/conda/issues/11775>).
Going to install micromamba to create environment ... installed at /home/sagemaker-user/.local/bin/micromamba
    Extracting and linking Conda environment ... done in 582 seconds.
2023-10-20 08:28:00.271 [1266/start/6982 (pid 1765)] Task is starting.
2023-10-20 08:28:00.271 1 task is running: start (1 running; 0 done).
2023-10-20 08:28:00.271 No tasks are waiting in the queue.
2023-10-20 08:28:00.271 3 steps have not started: train, end, multinode_end.
2023-10-20 08:28:04.728 [1266/start/6982 (pid 1765)] No dot env!
2023-10-20 08:28:09.959 [1266/start/6982 (pid 1765)] Task finished successfully.
2023-10-20 08:28:10.802 [1266/train/control-1266-start-6982 (pid 1791)] Task is starting.
2023-10-20 08:28:11.874 [1266/train/control-1266-start-6982 (pid 1791)] Traceback (most recent call last):
2023-10-20 08:28:11.878 [1266/train/control-1266-start-6982 (pid 1791)] File "/home/sagemaker-user/metaflow_demos/src/fsdp-multi-node-multi-gpu/fsdp_flow.py", line 30, in <module>
2023-10-20 08:28:11.878 [1266/train/control-1266-start-6982 (pid 1791)] @conda_base(
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] ^^^^^^^^^^^
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] File "/usr/local/lib/python3.11/site-packages/metaflow/decorators.py", line 418, in wrap
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] return _base_flow_decorator(decofunc, f, **kwargs)
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] File "/usr/local/lib/python3.11/site-packages/metaflow/decorators.py", line 408, in _base_flow_decorator
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] deco_instance = decofunc(attributes=kwargs, statically_defined=True)
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-20 08:28:11.879 [1266/train/control-1266-start-6982 (pid 1791)] File "/usr/local/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_decorator.py", line 301, in __init__
2023-10-20 08:28:11.880 [1266/train/control-1266-start-6982 (pid 1791)] super(CondaFlowDecorator, self).__init__(attributes, statically_defined)
2023-10-20 08:28:11.880 [1266/train/control-1266-start-6982 (pid 1791)] File "/usr/local/lib/python3.11/site-packages/metaflow/decorators.py", line 185, in __init__
2023-10-20 08:28:11.880 [1266/train/control-1266-start-6982 (pid 1791)] super(FlowDecorator, self).__init__(*args, **kwargs)
2023-10-20 08:28:11.993 [1266/train/control-1266-start-6982 (pid 1791)] File "/usr/local/lib/python3.11/site-packages/metaflow/decorators.py", line 123, in __init__
2023-10-20 08:28:11.994 [1266/train/control-1266-start-6982 (pid 1791)] raise InvalidDecoratorAttribute(self.name, k, self.defaults)
2023-10-20 08:28:11.994 [1266/train/control-1266-start-6982 (pid 1791)] metaflow.exception.InvalidDecoratorAttribute: Decorator 'conda_base' does not support the attribute 'channels'. These attributes are supported: packages, libraries, python, disabled.
2023-10-20 08:28:12.149 [1266/train/control-1266-start-6982 (pid 1791)] Task failed.
2023-10-20 08:28:12.307 Workflow failed.
2023-10-20 08:28:12.307 Terminating 0 active tasks...
2023-10-20 08:28:12.307 Flushing logs...
Probably need to clean up some stuff. It might be interference from the 
metaflow-ray
extension. Will do that tomorrow 🙂
d
Hun. That’s possible. If you have a small repro I can try. Also let me know where the ray extension you are using is (if it’s the one public I can look it up) and I can check for it. It’s definitely not loading the extension properly in your control task.
The control task does not seem to run in conda (which is the default behavior). But am not sure what it is running in.
a
Yay! It works now. Thanks a bunch Romain!
d
what was the issue?
a
I just used a deep learning image on Studio and installed Metaflow and the Metaflow extension from scratch. Maybe the 
metaflow-ray
package which was pre-baked inside the custom kernel image was interfering with the experimental metaflow extension package?
d
ah. Interesting. I looked at your ray extension and I would rename the 
__init__.py
file to something like 
mfextinit_ray.py
. That may be causing the issue
👀 1
the 
__init__.py
files don’t always play nice with namespace packages (which is what Metaflow Extensions uses).
a
Ahh ok will do! Thanks Romain!
d
depending on the loading order or something, it’s possible that it basically “ignored” the other extension.
a
Got it. Makes sense.
d
so ya, that simple change should make your extension play nice with other extensions 🙂
❤️ 1
the name can be anything as long as it starts with 
mfextinit_
👍 1
c
after the change to 
_init_.py
--> 
mfextinit_ray.py
the same workflows that were good before are producing:
Copy code
Traceback (most recent call last):
  File "/Users/eddie/Dev/metaflow-ray/examples/cloud-dependencies/ray_cpu_multinode.py", line 1, in <module>
    from metaflow import FlowSpec, step, metaflow_ray, batch, current
  File "/Users/eddie/mambaforge/envs/new-mf-ray/lib/python3.12/site-packages/metaflow/__init__.py", line 115, in <module>
    from .plugins.datatools import S3
  File "/Users/eddie/mambaforge/envs/new-mf-ray/lib/python3.12/site-packages/metaflow/plugins/__init__.py", line 132, in <module>
    STEP_DECORATORS = resolve_plugins("step_decorator")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eddie/mambaforge/envs/new-mf-ray/lib/python3.12/site-packages/metaflow/extension_support/plugins.py", line 144, in resolve_plugins
    raise ValueError(
ValueError: Cannot locate 'RayParallelDecorator' class for step_decorator plugin at 'metaflow_extensions.ray.plugins'
cc @acoustic-van-30942
d
let me check. It should work the same way. one sec
thankyou 2
oh right. ok, one sec.
could you give me write rights so I can push the fix?
👍 1
c
d
thx! Ok, could you run your tests again? It should fix it (at least the extension loads properly)
c
resolved the error! thanks
a
Thanks so much guys!
d
cool. That should avoid any clashes now. In general, you can have 
__init__.py
files but if you are sure that nothing else will be added to that directory when the namespace packages are installed (they are all installed in the same place in some cases).
I’ll add a warning to the import system too to warn against this.
(well, I’ll see how easy it is)
c
cool, appreciate the explanation. this pattern is easy enough to pivot to 🙂
d
yep. Just name all your init.py files mfextinit_. The reason it didn’t work is that the 
.RayDecorator
no longer pointed to anything since there was no 
__init__.py
🙂
👍 1
oh right. I wanted to mention this. By default a control node like the one you use (UBF_CONTROL) does not get any conda environment (it runs on whatever the machine has). I do have support to specify that the control node also gets the same environment as the worker nodes. I don’t have support (rn) to specify different environments for both types of nodes. If any of these are important/needed for you, let me know.
a
I think it is important. For some reason, I always thought the control node would be able to access the virtual env.
d
historically that hasn’t been the case the logic being that the control node is something outside of the user’s control and therefore the dependencies that they specify for their user code don’t necessarily apply to it.
a
Oh wait it might be because historically I always ran the flow within a virtual environment created from an 
env.yml
, and I stopped doing that recently.
That would make sense then. This is a good call out, thanks
d
If you want the contorl node to have the exact same set of dependencies (so the case I do support), you can do this. WARNING: this is NOT compatible with the mainline decorators and I may change some of these things (no plan to rn but this is more bleeding than bleeding edge 🙂: • for your 
metaflow_ray
decorator, instead of inheriting from just 
StepDecorator
, do this:
Copy code
from metaflow_extensions.netflix_ext.plugins.conda.conda_common_decorator import StepRequirementMixin
class RayParallelDecorator(StepRequirementMixin, ParallelDecorator):
• and then just add this method to your class:
Copy code
def default_disabled(self, ubf_context: str) -> Optional[bool]:
    return False
One way this may change (I don’t currently have a need for it but could evolve) is that instead of this method (or in addition), I would pass the requirements the user has to a function that would change it to something else based on 
ubf_context
. This is a bit hacky rn but it does work (we have a different ray integration but some of the issues were the same).
❤️ 1
👀 1
a
This is great! Thanks Romain. I'll create a PR and make the necessary adjustments.
d
(again, if you make this change, it won’t work without the Netflix Extensions which may limit the appeal/availability of the metaflow_ray decorator).
(ok, maybe I am being a bit heavy handed 🙂 )
a
Ah ok yeah, we probably don't wanna do that then. At our org, most people are just using their custom images when working with the 
metaflow_ray
decorator
10 Views
Open in Slack
PreviousNext
Powered by