https://outerbounds.com/ logo
Join Slack
Powered by
Thank you for sharing the Metaflow project with us...
# ask-metaflow
c
Thank you for sharing the Metaflow project with us. It is such an amazing tool! We are experimenting with running Metaflow in an AKS cluster with Argo as the workflow engine. We had been managing project dependencies (internal and external) with Poetry, but for this prototype we have switched to using Conda because that is what we are seeing in the docs as the supported pattern. We now are seeing: Setting up task environment. Downloading code package... Code package downloaded. Task is starting. Bootstrapping environment... Environment bootstrapped. With a simple flow of 4 steps that run instantly locally we are seeing runtime of about 3.5 minutes, just under a minute per step. We are thinking the majority of this time might be the bootstrapping of the environment where it is installing Conda packages but are not certain how to confirm this. If this is indeed the bottleneck, is there support for image caching for the container image that is doing the work so that it does not have to be built back up for each step? Our use case is for parameterized calculation pipelines that will often foreach out to around 500 different steps and join back together and for different customers with different data sets this workflow could be run 1000 times in a day with different inputs. If the image bootstrapping was 30 seconds, this means we would be spending 1000*500*30 seconds on this. What is the best practice in reducing some of this bootstrapping overhead? Thanks again!
✅ 1
a
With Argo the first time the flow runs can take a bit time but the following ones should be much faster to setup (we use AKS as well). What node specs does your cluster have?
d
baking a docker image will definitely get you the fastest performance. There is work ongoing to speed this up significantly (in collaboration with some of the folks from the Mamba project). Not ETA on it yet. There is a faster implementation out there (https://github.com/Netflix/metaflow-nflx-extensions) which I have been testing out (I will actually push out an update to this package shortly because I haven’t released it since 2.7.20 came out and it’s incompatible now — give me a few hours to do that; if you wnat to use it now, just use a metaflow version < 2.7.20) which makes things significantly faster. It’s not fully supported and may change a bit but it is the version we are testing out for deployment within Netflix.
c
@alert-florist-32940 We are using Standard_D4as_v5 for our Node Pool. @dry-beach-38304 Thank you so much for your work on this. Performance improvements always appreciated! It would be nice if we can get great performance without having to manage a custom image containing dependencies to pair with every flow that we develop.
d
Thr plugin I pointed you to should do that but it won’t be as performant as a custom docker obviously. That’s all I was getting at. It’s still much faster than the current version :). You can give it a try and let me know.
apologies for the delay. v0.0.3 was actually already compatible with 2.7.20+. I just added v0.0.4 which has just a bit of cleanup and I also updated the conda-lock PR because of the bug they have wrt pip+conda packages (only matters if you use a mix of pip and conda packages — if you use pure conda, this does not impact you).
Let me know how it goes for you.
c
@dry-beach-38304 I noticed that the install instructions in the readme say that only S3 is supported at this time. To add Azure support, would I just need to add a block of code for 
elif datastore_type == "azure"
here? https://github.com/Netflix/metaflow-nflx-extensions/blob/3445dfe20a93680126080a2af[…]a3249f8f/metaflow_extensions/netflix_ext/plugins/conda/utils.py
d
hey @cuddly-family-48123, I don’t run azure so didn’t add it but you probably have that one and add something similar here: https://github.com/Netflix/metaflow-nflx-extensions/blob/3445dfe20a93680126080a2af[…]/metaflow_extensions/netflix_ext/config/mfextinit_netflixext.py. If you are willing to test it out, I can create a branch for you to try it out. There is also probably a smallish modification here: https://github.com/Netflix/metaflow-nflx-extensions/blob/3445dfe20a93680126080a2af[…]aflow_extensions/netflix_ext/cmd/environment/environment_cmd.py
let me do that for you.
c
Thanks @dry-beach-38304. I can try it out.
d
gimme a few minutes, i’ll push out a new package for you to try.
👍 1
ok, v0.0.5 is publishing. that should have azure support
all done. let me know how it goes and what works — and more importantly what doesn’t work — for you.
(note to get the maximum speed, make sure you set 
CONDA_PREFERRED_FORMAT
to 
.conda
)
c
Thanks! I'll take a look at getting it installed and trying it out.
thanks 1
@dry-beach-38304 I got this error, but it looks like it can be fixed by replacing 
"CONFIG_%sROOT"
with 
"CONDA_%sROOT"
on line 77 of 
/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials/param-ex$ python simple_conda.py  --environment=conda run
Metaflow 2.7.22+netflix-ext(0.0.5) executing SimpleConda for user:bbrandt
    Internal error
Traceback (most recent call last):
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 1009, in start
    ctx.obj.environment.validate_environment(echo, datastore)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 169, in validate_environment
    self._conda = Conda(echo, datastore_type)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 145, in __init__
    get_conda_root(self._datastore_type)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py", line 77, in get_conda_root
    conda_root = getattr(mf_config, "CONFIG_%sROOT" % datastore_type.upper())
AttributeError: module 'metaflow.metaflow_config' has no attribute 'CONFIG_AZUREROOT'
After that fix this is the next error. I am taking a break for the weekend and will pick this up again on Monday afternoon (CST). Thanks for your help thus far!
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$  cd /home/bbrandt/src/tutorials ; /usr/bin/env /home/bbrandt/mambaforge/envs/intro-to-mf/bin/python /home/bbrandt/.vscode-server/extensions/ms-python.python-2023.2.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 57631 -- /home/bbrandt/src/tutorials/param-ex/simple_conda.py --environment=conda run 
Metaflow 2.7.22+netflix-ext(0.0.5) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Internal error
Traceback (most recent call last):
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 828, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 877, in before_run
    decorators._init_step_decorators(
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/decorators.py", line 541, in _init_step_decorators
    deco.step_init(
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py", line 195, in step_init
    self._base_attributes = self._get_base_attributes()
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py", line 346, in _get_base_attributes
    return self._flow._flow_decorators["conda_base"].attributes
AttributeError: 'list' object has no attribute 'attributes'
I think this may be a clue for us: `# _flow_decorators. _flow_decorators is of type `{key:[decos]}`` https://github.com/Netflix/metaflow/blob/2f9e443574de038a40f9af0d07b6fc71327dce8b/metaflow/decorators.py#L404
d
Oh. Thank you. Let me fix both of those. Right. The flow decorator changed recently in the last version or so and I forgot to update to thst. Apologies. I will have a fix for both of those tonight.
I fixed it on my internal version and forgot to fix it there.
ok, sorry about that — now fixed. It will require 2.7.22 but I guess that’s what you had already since you were hitting that conda_base thing (flow decorators were made into list in that version).
thanks for being an alpha tester! Let me know if you run into more issues. I’ll also do another pass over the weekend to make sure that both this version and the one I am testing internally are in sync… Would avoid things like this so I can try to make sure it works properly 🙂
c
Thanks for the fixes! I've run 
pip install metaflow-netflixext --upgrade
and trying it out now. Next error I ran into was:
Copy code
Internal error:
    Azure error: Operation returned an invalid status 'This request is not authorized to perform this operation using this permission.'
    ErrorCode:AuthorizationPermissionMismatch
But that just required adding the 
Storage Blob Data Contributor Role
to my service principal account I was running under. Kind of like this: https://stackoverflow.com/a/52770419
d
oh that one I am not sure. Do you happen to have the full trace? I can ask around. I don’t use azure here but can hopefully find someone who knows.
ah sorry you solved it
c
Yes. I posted the solution there as well for others to reference...
d
where was it coming from (ie: do you know which call metaflow was making). I wouldn’t have expected it to make different calls compared to what it does for the artifacts.
and apart from that — is it working ok now?
c
Yes, it's odd that I would not have needed the permission before. Maybe it was just silently failing previously with the old version.
Next issue I am looking at now is:
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials/param-ex$ python simple_conda.py  --environment=conda run
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
    Internal error
Traceback (most recent call last):
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 828, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/cli.py", line 886, in before_run
    obj.package = MetaflowPackage(
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 120, in init_environment
    self._resolve_environments(echo, need_resolution)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 242, in _resolve_environments
    env_id, resolved_env = f.result()
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(self.args, self.kwargs)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 203, in _resolve
    self._conda.resolve(
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 196, in resolve
    packages = self._resolve_env_with_conda(deps, sources, architecture)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1269, in _resolve_env_with_conda
    conda_result = json.loads(self._call_conda(args, addl_env=addl_env))
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 3530 column 1 (char 123854)
Debugging through it, at line 1269 of metaflow_extensions/netflix_ext/plugins/conda/conda.py it looks like the output has extra text after the JSON:
Copy code
"dry_run": true,
  "prefix": "/tmp/tmpajx9t8sp/prefix",
  "success": true
}
Dry run. Exiting.'
d
oh, that looks like mamba spitting out some jibberish.
ah, gah.
of fcourse it does
c
Here's the full output from running mamba.
call-conda-output.log
d
ya, it’s that last line. Let me see if I can figure out how to make it not output that last line.
c
Yep. That's what I'm looking for now as well.
d
you would think that if you tell it to output json, well, it would output json 🙂
😁 1
ok, try this: on line 2124, remove 
stderr=subprocess.STDOUT
of 
conda.py
it actually just prints it in the stderr and I was capturing both.
c
That fixed that issue. Now I see:
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials/param-ex$ python simple_conda.py  --environment=conda run
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
Dry run. Exiting.
 done in 24 seconds.
    Downloading 2(web) + 0(cache) packages ... done in 13 seconds.
    Transmuting 59 packages ... done in 0 seconds.
Got the following errors while loading packages:
Error transmuting 'six-1.16.0-pyh6c4a22f_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libsqlite-3.40.0-h753d276_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libstdcxx-ng-12.2.0-h46fd767_19': Requesting to transmute package without cph or micromamba
Error transmuting 'libev-4.33-h516909a_1': Requesting to transmute package without cph or micromamba
Error transmuting 'pandas-1.4.4-py310h769672d_0': Requesting to transmute package without cph or micromamba
Error transmuting 'gflags-2.2.2-he1b5a44_1004': Requesting to transmute package without cph or micromamba
Error transmuting 'libevent-2.1.10-h28343ad_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libgcc-ng-12.2.0-h65d4601_19': Requesting to transmute package without cph or micromamba
Error transmuting 'libopenblas-0.3.21-pthreads_h78a6416_3': Requesting to transmute package without cph or micromamba
Error transmuting 'ncurses-6.3-h27087fc_1': Requesting to transmute package without cph or micromamba
Error transmuting 'pycparser-2.21-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'isodate-0.6.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libssh2-1.10.0-hf14f497_3': Requesting to transmute package without cph or micromamba
Error transmuting 'oauthlib-3.2.2-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'blinker-1.5-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'glog-0.6.0-h6f12383_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libcrc32c-1.1.2-h9c3ff4c_0': Requesting to transmute package without cph or micromamba
Error transmuting 'readline-8.1.2-h0f457ee_0': Requesting to transmute package without cph or micromamba
Error transmuting '_libgcc_mutex-0.1-conda_forge': Requesting to transmute package without cph or micromamba
Error transmuting 'libnsl-2.0.0-h7f98852_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libthrift-0.16.0-he500d00_2': Requesting to transmute package without cph or micromamba
Error transmuting 'brotlipy-0.7.0-py310h5764c6d_1005': Requesting to transmute package without cph or micromamba
Error transmuting 'bzip2-1.0.8-h7f98852_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libcblas-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'requests-oauthlib-1.3.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgfortran-ng-12.2.0-h69a702a_19': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlicommon-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'python-dateutil-2.8.2-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'pysocks-1.7.1-pyha2e5f31_6': Requesting to transmute package without cph or micromamba
Error transmuting 'charset-normalizer-2.1.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'typing_extensions-4.4.0-pyha770c72_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgomp-12.2.0-h65d4601_19': Requesting to transmute package without cph or micromamba
Error transmuting 'zlib-1.2.13-h166bdaf_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libzlib-1.2.13-h166bdaf_4': Requesting to transmute package without cph or micromamba
Error transmuting 'liblapack-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'libuuid-2.32.1-h7f98852_1000': Requesting to transmute package without cph or micromamba
Error transmuting 'libgfortran5-12.2.0-h337968e_19': Requesting to transmute package without cph or micromamba
Error transmuting 'libblas-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'msal_extensions-1.0.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'keyutils-1.6.1-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'tk-8.6.12-h27826a3_0': Requesting to transmute package without cph or micromamba
Error transmuting 'xz-5.2.6-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'idna-3.4-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'ucx-1.12.1-h7a399c7_1': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlidec-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'libffi-3.4.2-h7f98852_5': Requesting to transmute package without cph or micromamba
Error transmuting 'c-ares-1.18.1-h7f98852_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlienc-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'wheel-0.38.4-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'snappy-1.1.9-hbd366e4_2': Requesting to transmute package without cph or micromamba
Error transmuting 'libedit-3.1.20191231-he28a2e2_2': Requesting to transmute package without cph or micromamba
Error transmuting '_openmp_mutex-4.5-2_gnu': Requesting to transmute package without cph or micromamba
Error transmuting 'libutf8proc-2.8.0-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'pyjwt-2.6.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'typing-extensions-4.4.0-hd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'msrest-0.7.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'parquet-cpp-1.5.1-2': Requesting to transmute package without cph or micromamba
Error transmuting 'azure-identity-1.12.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'azure-storage-blob-12.14.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
    Conda ran into an error while setting up environment.:
    Could not fetch packages -- see pretty-printed errors above.
Is that something I missed in the setup instructions?
d
yes, if you specify 
.conda
as the packages you want and it doesn’t find one, it will transmute it for you but you need either 
cph
or 
micromamba
installed.
c
Doing this now:
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials/param-ex$ cph
Command 'cph' not found, but can be installed with:
sudo apt install conda-package-handling
d
it’s also a conda package so you can also install it that way but yes, you need to have it installed
and that’s the one.
(I’ve not installed it with apt-get but hopefully should do the right thing)
it may be an older version though — that’s my only concern but let me know which version it installs and I can tell you.
micromamba also works (much lighter weight)
(that’s just a binary with no deps)
I’ll clarify the README about cph/micromamba.
c
Ah, ok. If I have conda and mamba installed would I just install micromamba beside those?
d
yep
if it finds the other two, it should find that one 🙂
✅ 1
c
If you are updating the readme, here's a link for the micromamba install: https://mamba.readthedocs.io/en/latest/installation.html#micromamba
I cancelled the apt install and will try micromamba.
d
on a remote node, it will probably install it as well to actually setup the env (hopefully that part works too): https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/conda.py#L1616
c
Great. I'm trying local run now and can run in argo if that works.
Odd. I have micromamba in the path, but getting the same error even after restarting VSCode and my terminal.
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ python param-ex/simple_conda.py  --environment=conda run
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
Dry run. Exiting.
 done in 25 seconds.
    Transmuting 59 packages ... done in 3 seconds.
Got the following errors while loading packages:
Error transmuting '_libgcc_mutex-0.1-conda_forge': Requesting to transmute package without cph or micromamba
Error transmuting 'libcrc32c-1.1.2-h9c3ff4c_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgfortran-ng-12.2.0-h69a702a_19': Requesting to transmute package without cph or micromamba
Error transmuting '_openmp_mutex-4.5-2_gnu': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlicommon-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'libnsl-2.0.0-h7f98852_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libffi-3.4.2-h7f98852_5': Requesting to transmute package without cph or micromamba
Error transmuting 'libev-4.33-h516909a_1': Requesting to transmute package without cph or micromamba
Error transmuting 'gflags-2.2.2-he1b5a44_1004': Requesting to transmute package without cph or micromamba
Error transmuting 'libuuid-2.32.1-h7f98852_1000': Requesting to transmute package without cph or micromamba
Error transmuting 'libzlib-1.2.13-h166bdaf_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libutf8proc-2.8.0-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'keyutils-1.6.1-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'snappy-1.1.9-hbd366e4_2': Requesting to transmute package without cph or micromamba
Error transmuting 'c-ares-1.18.1-h7f98852_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libblas-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlidec-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'glog-0.6.0-h6f12383_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgomp-12.2.0-h65d4601_19': Requesting to transmute package without cph or micromamba
Error transmuting 'libedit-3.1.20191231-he28a2e2_2': Requesting to transmute package without cph or micromamba
Error transmuting 'bzip2-1.0.8-h7f98852_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libbrotlienc-1.0.9-h166bdaf_8': Requesting to transmute package without cph or micromamba
Error transmuting 'libcblas-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'zlib-1.2.13-h166bdaf_4': Requesting to transmute package without cph or micromamba
Error transmuting 'liblapack-3.9.0-16_linux64_openblas': Requesting to transmute package without cph or micromamba
Error transmuting 'blinker-1.5-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'readline-8.1.2-h0f457ee_0': Requesting to transmute package without cph or micromamba
Error transmuting 'charset-normalizer-2.1.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'idna-3.4-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgcc-ng-12.2.0-h65d4601_19': Requesting to transmute package without cph or micromamba
Error transmuting 'pyjwt-2.6.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'pysocks-1.7.1-pyha2e5f31_6': Requesting to transmute package without cph or micromamba
Error transmuting 'xz-5.2.6-h166bdaf_0': Requesting to transmute package without cph or micromamba
Error transmuting 'six-1.16.0-pyh6c4a22f_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libssh2-1.10.0-hf14f497_3': Requesting to transmute package without cph or micromamba
Error transmuting 'typing_extensions-4.4.0-pyha770c72_0': Requesting to transmute package without cph or micromamba
Error transmuting 'typing-extensions-4.4.0-hd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'wheel-0.38.4-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'isodate-0.6.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libsqlite-3.40.0-h753d276_0': Requesting to transmute package without cph or micromamba
Error transmuting 'msal_extensions-1.0.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'requests-oauthlib-1.3.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'python-dateutil-2.8.2-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'parquet-cpp-1.5.1-2': Requesting to transmute package without cph or micromamba
Error transmuting 'pycparser-2.21-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'msrest-0.7.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'brotlipy-0.7.0-py310h5764c6d_1005': Requesting to transmute package without cph or micromamba
Error transmuting 'oauthlib-3.2.2-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'azure-identity-1.12.0-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libgfortran5-12.2.0-h337968e_19': Requesting to transmute package without cph or micromamba
Error transmuting 'azure-storage-blob-12.14.1-pyhd8ed1ab_0': Requesting to transmute package without cph or micromamba
Error transmuting 'libevent-2.1.10-h28343ad_4': Requesting to transmute package without cph or micromamba
Error transmuting 'libstdcxx-ng-12.2.0-h46fd767_19': Requesting to transmute package without cph or micromamba
Error transmuting 'tk-8.6.12-h27826a3_0': Requesting to transmute package without cph or micromamba
Error transmuting 'ncurses-6.3-h27087fc_1': Requesting to transmute package without cph or micromamba
Error transmuting 'libthrift-0.16.0-he500d00_2': Requesting to transmute package without cph or micromamba
Error transmuting 'libopenblas-0.3.21-pthreads_h78a6416_3': Requesting to transmute package without cph or micromamba
Error transmuting 'ucx-1.12.1-h7a399c7_1': Requesting to transmute package without cph or micromamba
Error transmuting 'pandas-1.4.4-py310h769672d_0': Requesting to transmute package without cph or micromamba
    Conda ran into an error while setting up environment.:
    Could not fetch packages -- see pretty-printed errors above.

(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ micromamba

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

Version: 1.3.1

Usage: /home/bbrandt/.local/bin/micromamba [OPTIONS] [SUBCOMMAND]

Options:
  -h,--help                   Print this help message and exit
  --version                   


Configuration options:
  --rc-file TEXT ...          Paths to the configuration files to use
  --no-rc                     Disable the use of configuration files
  --no-env                    Disable the use of environment variables


Global options:
  -v,--verbose                Set verbosity (higher verbosity with multiple -v, e.g. -vvv)
  --log-level ENUM:value in {critical->5,debug->1,error->4,info->2,off->6,trace->0,warning->3} OR {5,1,4,2,6,0,3}
                              Set the log level
  -q,--quiet                  Set quiet mode (print less output)
  -y,--yes                    Automatically answer yes on prompted questions
  --json                      Report all output as json
  --offline                   Force use cached repodata
  --dry-run                   Only display what would have been done
  --download-only             Only download and extract packages, do not link them into environment.
  --experimental              Enable experimental features


Prefix options:
  -r,--root-prefix TEXT       Path to the root prefix
  -p,--prefix TEXT            Path to the target prefix
  -n,--name TEXT              Name of the target prefix

Subcommands:
  shell                       Generate shell init scripts
  create                      Create new environment
  install                     Install packages in active environment
  update                      Update packages in active environment
  self-update                 Update micromamba
  repoquery                   Find and analyze packages in active environment or channels
  remove                      Remove packages from active environment
  list                        List packages in active environment
  package                     Extract a package or bundle files into an archive
  clean                       Clean package cache
  config                      Configuration of micromamba
  info                        Information about micromamba
  constructor                 Commands to support using micromamba in constructor
  env                         List environments
  activate                    Activate an environment
  run                         Run an executable in an environment
  ps                          Show, inspect or kill running processes
  auth                        Login or logout of a given host
  search                      Find packages in active environment or channels

(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$
d
i am an idiot. my gosh
line 861 of conda.py, change 
if
to 
elif
.
c
Ah, that would explain how I got past the error the 2nd time I ran it... The files were transmuted by the 1st run that errored. 🙂
I unexpectedly started getting this error and wasn't sure what I did different.
Copy code
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
Dry run. Exiting.
 done in 23 seconds.
    Caching 226 items to azure ...    Failed to access Azure resource:
    The specified container does not exist.
    RequestId:20418be3-801e-0005-10e8-3ffd6c000000
    Time:2023-02-13T20:21:28.4536285Z
    ErrorCode:ContainerNotFound
    Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.
    RequestId:20418be3-801e-0005-10e8-3ffd6c000000
    Time:2023-02-13T20:21:28.4536285Z</Message></Error>
For that I just need to pre-create my Azure Blob container I configured with the 
METAFLOW_CONDA_AZUREROOT
config.
d
ya. I am not sure of all the intricacies of azure and how it works for that. For S3, I use the same bucket and it doesn’t matter if the prefix exists or not. Looks like it got further though if it is caching.
✅ 1
c
I'm not sure about all the error output, but it did run to completion now.
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ python param-ex/simple_conda.py  --environment=conda run
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
Dry run. Exiting.
 done in 23 seconds.
    Caching 226 items to azure ... done in 42 seconds.
    Caching 1 environment to azure ... done in 0 seconds.
2023-02-13 14:28:59.254 Workflow starting (run-id 34):
2023-02-13 14:29:00.874 Falling back to 'mamba' to resolve as conda-lock not installed
2023-02-13 14:29:01.195 Creating Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)...
2023-02-13 14:29:02.192     Extracting and linking Conda environment .../tmp/mambaffCI6aBINfj: line 3: actions:: command not found
/tmp/mambaffCI6aBINfj: line 4: print:: command not found
/tmp/mambaffCI6aBINfj: line 5: __mamba_exe: command not found
/tmp/mambaffCI6aBINfj: line 5: __mamba_exe: command not found
/tmp/mambaffCI6aBINfj: line 5: \"${(@s: :)${(@s: :)COMP_LINE}:1}\": bad substitution
/tmp/mambaffCI6aBINfj: line 5: __mamba_exe: command not found
/tmp/mambaffCI6aBINfj: line 5: \n# Copyright (C) 2012 Anaconda, Inc\n# SPDX-License-Identifier: BSD-3-Clause\n\n__mamba_exe() (\n    "/home/bbrandt/.local/bin/micromamba" ""\n)\n\n__mamba_hashr() {\n    if [ -n "" ]; then\n        \rehash\n    elif [ -n "" ]; then\n        :  # pass\n    else\n        \hash -r\n    fi\n}\n\n__mamba_activate() {\n    \local ask_conda\n    ask_conda="" || \return\n    \eval ""\n    __mamba_hashr\n}\n\n__mamba_reactivate() {\n    \local ask_conda\n    ask_conda="" || \return\n    \eval ""\n    __mamba_hashr\n}\n\nmicromamba() {\n    \local cmd="__missing__"\n    case "" in\n        activate|deactivate)\n            __mamba_activate ""\n            ;;\n        install|update|upgrade|remove|uninstall)\n            __mamba_exe "" || \return\n            __mamba_reactivate\n            ;;\n        self-update)\n            __mamba_exe "" || \return\n\n            # remove leftover backup file on Windows\n            if [ -f "/home/bbrandt/.local/bin/micromamba.bkup" ]; then\n                rm -f /home/bbrandt/.local/bin/micromamba.bkup\n            fi\n            ;;\n        *)\n            __mamba_exe ""\n            ;;\n    esac\n}\n\nif [ -z "x" ]; then\n    \export CONDA_SHLVL=0\n    # In dev-mode MAMBA_EXE is python.exe and on Windows\n    # it is in a different relative location to condabin.\n    if [ -n "x" ] && [ -n "" ]; then\n        PATH="/home/bbrandt/micromamba/condabin:/home/bbrandt/mambaforge/envs/metaflow_c5b6dc01d4f4a08e54fef899876a247da78dfef5_f9c27900b0dfa3765069e94f41ccb3fcf1b9752f/bin:/home/bbrandt/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/bin/remote-cli:/home/bbrandt/.local/bin:/home/bbrandt/mambaforge/envs/intro-to-mf/bin:/home/bbrandt/mambaforge/condabin:/home/bbrandt/.nvm/versions/node/v18.13.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Volta:/mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/mnt/c/Program Files/dotnet:/mnt/c/Program Files (x86)/Microsoft SQL Server/160/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/160/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/160/DTS/Binn:/mnt/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files (x86)/GitExtensions:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Volta/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/Ben.Brandt/.dotnet/tools:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/dapr:/mnt/c/Users/Ben.Brandt/AppData/Roaming/Code/User/globalStorage/ms-vscode-remote.remote-containers/cli-bin:/mnt/c/Users/Ben.Brandt/.dapr/bin:/mnt/c/Users/Ben.Brandt/.dapr/bin:/mnt/c/Program Files (x86)/Rico Suter/NSwagStudio:/snap/bin"\n    else\n        PATH="/home/bbrandt/micromamba/condabin:/home/bbrandt/mambaforge/envs/metaflow_c5b6dc01d4f4a08e54fef899876a247da78dfef5_f9c27900b0dfa3765069e94f41ccb3fcf1b9752f/bin:/home/bbrandt/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/bin/remote-cli:/home/bbrandt/.local/bin:/home/bbrandt/mambaforge/envs/intro-to-mf/bin:/home/bbrandt/mambaforge/condabin:/home/bbrandt/.nvm/versions/node/v18.13.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Volta:/mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/mnt/c/Program Files/dotnet:/mnt/c/Program Files (x86)/Microsoft SQL Server/160/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/160/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/160/DTS/Binn:/mnt/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files (x86)/GitExtensions:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Volta/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/Ben.Brandt/.dotnet/tools:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/Users/Ben.Brandt/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/dapr:/mnt/c/Users/Ben.Brandt/AppData/Roaming/Code/User/globalStorage/ms-vscode-remote.remote-containers/cli-bin:/mnt/c/Users/Ben.Brandt/.dapr/bin:/mnt/c/Users/Ben.Brandt/.dapr/bin:/mnt/c/Program Files (x86)/Rico Suter/NSwagStudio:/snap/bin"\n    fi\n    \export PATH\n\n    # We're not allowing PS1 to be unbound. It must at least be set.\n    # However, we're not exporting it, which can cause problems when starting a second shell\n    # via a first shell (i.e. starting zsh from bash).\n    if [ -z "" ]; then\n        PS1=\n    fi\nfi\n\nif [ -n "" ]; then\n  if ! command -v compinit > /dev/null; then\n    autoload -U +X compinit && if [[ "" = true ]]; then\n      compinit -u\n    else\n      compinit\n    fi\n  fi\n  autoload -U +X bashcompinit && bashcompinit\n\n  _umamba_zsh_completions()\n  {\n    COMPREPLY=()\n  }\n\n  complete -o default -F _umamba_zsh_completions micromamba\nfi\nif [ -n "x" ]; then\n  _umamba_bash_completions()\n  {\n    COMPREPLY=()\n  }\n  complete -o default -F _umamba_bash_completions micromamba\nfi\n\n: File name too long
/tmp/mambaffCI6aBINfj: line 6: ]: command not found
/tmp/mambaffCI6aBINfj: line 7: },: command not found
/tmp/mambaffCI6aBINfj: line 8: context:: command not found
/tmp/mambaffCI6aBINfj: line 9: shell_type:: command not found
/tmp/mambaffCI6aBINfj: line 10: },: command not found
/tmp/mambaffCI6aBINfj: line 11: operation:: command not found
/tmp/mambaffCI6aBINfj: line 12: success:: command not found
critical libmamba Shell not initialized
By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): <https://docs.nvidia.com/cuda/eula/index.html>
 done in 19 seconds.
2023-02-13 14:29:21.975 [34/start/116 (pid 20111)] Task is starting.
2023-02-13 14:29:26.032 [34/start/116 (pid 20111)] alpha is 0.010000
2023-02-13 14:29:28.660 [34/start/116 (pid 20111)] Task finished successfully.
2023-02-13 14:29:29.900 Using existing Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)
2023-02-13 14:29:29.902 [34/pd_dates/117 (pid 20197)] Task is starting.
2023-02-13 14:29:34.359 [34/pd_dates/117 (pid 20197)] a
2023-02-13 14:29:36.123 [34/pd_dates/117 (pid 20197)] 0  42
2023-02-13 14:29:36.961 [34/pd_dates/117 (pid 20197)] Task finished successfully.
2023-02-13 14:29:38.197 Using existing Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)
2023-02-13 14:29:38.199 [34/end/118 (pid 20291)] Task is starting.
2023-02-13 14:29:42.159 [34/end/118 (pid 20291)] alpha is still 0.010000
2023-02-13 14:29:44.646 [34/end/118 (pid 20291)] Task finished successfully.
2023-02-13 14:29:44.996 Done!
d
let me know when you are satisfied with it — I’ll then make another release fixing (so far): • more docs around micromamba/cph and conda packages • the dry-run (capturing stderr) thing • the if/elif idiotness • If you want to update the README for the Azure specific things needed, that would be awesome too and I can merge that
ooh, that does not look like any of my code (those weird things). Let me look a bit moer at the output.
c
I think it's related to silliness in my Windows Subsystem for Linux/Ubuntu environment. May be a Windows executable that something is calling out to or else my system path has become too long with some of my recent installs.
d
ok, can you confirm you are executing in the right environment? (ie: it’s not silently not installing the conda env)?
btw, you probably figured this out already but you can tell what metaflow is calling in terms of conda stuff by running with 
METAFLOW_DEBUG_CONDA=1
. That should give you the command that it is executing to extract the conda environment which is probably what is causing this weirdness in the first place
c
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ which conda
/home/bbrandt/mambaforge/condabin/conda
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ which mamba
/home/bbrandt/mambaforge/condabin/mamba
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ which micromamba
/home/bbrandt/.local/bin/micromamba
I have not figured that out, but will try it now.
d
sorry, I meant in your step, that it is indeed executing within the environment you configured (like print the version of the package that is installed to confirm it’s the one you want).
when you re-run, it’ll probably not recreate the envs since they exist so you may have to thrash them to get it to do it again.
c
Output looks great on 2nd run and was much quicker:
Copy code
Metaflow 2.7.22+netflix-ext(0.0.6) executing SimpleConda for user:bbrandt
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
Falling back to 'mamba' to resolve as conda-lock not installed
    All items already cached in azure.
2023-02-13 14:36:22.868 Workflow starting (run-id 35):
2023-02-13 14:36:24.486 Falling back to 'mamba' to resolve as conda-lock not installed
2023-02-13 14:36:24.816 Using existing Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)
2023-02-13 14:36:24.817 [35/start/120 (pid 21342)] Task is starting.
2023-02-13 14:36:28.750 [35/start/120 (pid 21342)] alpha is 0.010000
2023-02-13 14:36:31.257 [35/start/120 (pid 21342)] Task finished successfully.
2023-02-13 14:36:32.486 Using existing Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)
2023-02-13 14:36:32.487 [35/pd_dates/121 (pid 21397)] Task is starting.
2023-02-13 14:36:36.575 [35/pd_dates/121 (pid 21397)] a
2023-02-13 14:36:38.345 [35/pd_dates/121 (pid 21397)] 0  42
2023-02-13 14:36:39.204 [35/pd_dates/121 (pid 21397)] Task finished successfully.
2023-02-13 14:36:40.498 Using existing Conda environment c5b6dc01d4f4a08e54fef899876a247da78dfef5 (f9c27900b0dfa3765069e94f41ccb3fcf1b9752f)
2023-02-13 14:36:40.499 [35/end/122 (pid 21508)] Task is starting.
2023-02-13 14:36:44.478 [35/end/122 (pid 21508)] alpha is still 0.010000
2023-02-13 14:36:46.956 [35/end/122 (pid 21508)] Task finished successfully.
2023-02-13 14:36:47.320 Done!
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$
d
yep, it’s using the cached environments now.
c
Yes. I can delete the blob from storage to reproduce the 1st time run.
d
sorry, I meant trash the local ones
don’t trash blobs in azure
c
Ah, local, meaning my conda environment I created or different local?
d
if you list your environments, you should have some that look like 
metaflow_c5b6dc01d4f4a08e54fef899876a247da78dfef5_f9c27900b0dfa3765069e94f41ccb3fcf1b9752f
or something like that.
you can remove those environments using the normal conda tools
that way, when you re-reun, it won’t reresolve your enviornments but will instead just recreate the environment locally.
(and in general, avoid trashing the blob storage unless you also trash the 
~/.metaflow/condav2.cnd
file else it will get confused as to where things are cached 🙂 )
c
Got it! Yes the local envs showed up with 
mamba env list
https://outerbounds-community.slack.com/archives/C02116BBNTU/p1676320306927909?thread_ts=1675972930.806409&amp;cid=C02116BBNTU I am good with your list of updates. Azure Specific items for the readme: • Manually create the blob container specified in the 
METAFLOW_CONDA_AZUREROOT
• Grant the 
Storage Blob Data Contributor
Role on the Storage Account to the Service Principal or User Accounts that will be accessing (https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal#assign-an-azure-role)
d
ok cool. I’ll do that later tonight. Let me know if it speeds it up enough for you. Hopefully it’s somewhat of an improvement (modulo the bugs you found — thank you very much for your patience on those).
👍 1
c
d
interesting. Is your 
micromamba
executable the 
micromamba.sh
?
for me, the executable is directly a binary and it doesn’t do any 
activate
stuff which it seems is what is triggering this.
c
Copy code
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ which micromamba
/home/bbrandt/.local/bin/micromamba
maybe?
d
right, and 
file /home/bbrandt/.local/bin/micromamba
or 
head /home/bbrandt/.local/bin/micromamba
?
c
(intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ file /home/bbrandt/.local/bin/micromamba /home/bbrandt/.local/bin/micromamba: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, stripped (intro-to-mf) bbrandt@939TQ13:~/src/tutorials$ head /home/bbrandt/.local/bin/micromamba ELF>�+@��@8 @@@@�����F%�F%P%P%P%�A��A�������]�-]�-�i��i��i�� �l��l��l�p��� �i��i��i�P�tdX��X��X���_�_Q�tdR�td�i��i��/lib64/ld-linux-x86-64.so.2GNU��Th j9p.�...
I guess not.
d
ah crap, it is the binary. why the crap is it complaining so much then.
I’ll dig into it in a bit to see if I can squelsh that.
oh wait, does your shell init script have something in it that micromamba acdded?
c
could be something like that...
Copy code
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/bbrandt/mambaforge/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/bbrandt/mambaforge/etc/profile.d/conda.sh" ]; then
        . "/home/bbrandt/mambaforge/etc/profile.d/conda.sh"
    else
        export PATH="/home/bbrandt/mambaforge/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "/home/bbrandt/mambaforge/etc/profile.d/mamba.sh" ]; then
    . "/home/bbrandt/mambaforge/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<


# >>> mamba initialize >>>
# !! Contents within this block are managed by 'mamba init' !!
export MAMBA_EXE="/home/bbrandt/.local/bin/micromamba";
export MAMBA_ROOT_PREFIX="/home/bbrandt/micromamba";
__mamba_setup="$("$MAMBA_EXE" shell hook --shell bash --prefix "$MAMBA_ROOT_PREFIX" 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__mamba_setup"
else
    if [ -f "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh" ]; then
        . "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh"
    else
        export  PATH="/home/bbrandt/micromamba/bin:$PATH"  # extra space after export prevents interference from conda init
    fi
fi
unset __mamba_setup
# <<< mamba initialize <<<
in my .bashrc
So, maybe something with that 
/home/bbrandt/micromamba/etc/profile.d/micromamba.sh
d
ya, I am guessing that micromamba.sh is what is causing the issues (as you seem to have pointed out in your log).
you can try moving that file aside for now and see if that helps. On my end, I’ll try adding that file and figuring out if I can it to work with that file present.
c
Looks like in my .bashrc it's actually going into the section where I have 
echo running MAMBA_EXE shell hook
instead of the micromamba.sh.
Copy code
# >>> mamba initialize >>>
# !! Contents within this block are managed by 'mamba init' !!
export MAMBA_EXE="/home/bbrandt/.local/bin/micromamba";
export MAMBA_ROOT_PREFIX="/home/bbrandt/micromamba";
__mamba_setup="$("$MAMBA_EXE" shell hook --shell bash --prefix "$MAMBA_ROOT_PREFIX" 2> /dev/null)"
if [ $? -eq 0 ]; then
    echo running MAMBA_EXE shell hook
    eval "$__mamba_setup"
else
    if [ -f "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh" ]; then
        . "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh"
    else
        export  PATH="/home/bbrandt/micromamba/bin:$PATH"  # extra space after export prevents interference from conda init
    fi
fi
unset __mamba_setup
# <<< mamba initialize <<<
But even if I comment out this section:
Copy code
# >>> mamba initialize >>>
# !! Contents within this block are managed by 'mamba init' !!
export MAMBA_EXE="/home/bbrandt/.local/bin/micromamba";
export MAMBA_ROOT_PREFIX="/home/bbrandt/micromamba";
# __mamba_setup="$("$MAMBA_EXE" shell hook --shell bash --prefix "$MAMBA_ROOT_PREFIX" 2> /dev/null)"
# if [ $? -eq 0 ]; then
#     echo running MAMBA_EXE shell hook
#     eval "$__mamba_setup"
# else
    if [ -f "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh" ]; then
        . "/home/bbrandt/micromamba/etc/profile.d/micromamba.sh"
    else
        export  PATH="/home/bbrandt/micromamba/bin:$PATH"  # extra space after export prevents interference from conda init
    fi
# fi
# unset __mamba_setup
# <<< mamba initialize <<<
and move micromamba.sh out of the way, I am still able to reproduce the issue. But micromamba is stated as being experimental and it doesn't stop your extension or metaflow from working, so I'm going to ignore it for now and move on. 🙂
👍 1
d
I’ll try to improve for you but glad to know it’s not a blocker.
👍 1
let me know if you would like to see anything else. As you can tell, it’s still being developed so if you don’t like how it does something or would like it to do something differently, don’t hesitate to let me know.
c
Thanks!
d
if you have time @cuddly-family-48123, I would greatly appreciate a quick look at https://github.com/Netflix/metaflow-nflx-extensions/pull/3 to make sure I captured all the right fixes/readme changes. Thanks!
c
LGTM!
d
ty. WIll merge and update. Let me know if you need anything else.
c
@dry-beach-38304 I was able to test on Argo in Azure and I am seeing this exception:
Copy code
Bootstrapping environment ...
    Setting up Conda ... done in 0 seconds.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 62, in <module>
    bootstrap_environment(*sys.argv[1:])
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py", line 32, in bootstrap_environment
    cached_info = read_conda_manifest(DATASTORE_LOCAL_DIR)
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 827, in read_conda_manifest
    return CachedEnvironmentInfo.from_dict(json.load(f))
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 797, in from_dict
    resolved_env = ResolvedEnvironment.from_dict(
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 626, in from_dict
    all_packages = [PackageSpecification.from_dict(pd) for pd in d["packages"]]
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 626, in <listcomp>
    all_packages = [PackageSpecification.from_dict(pd) for pd in d["packages"]]
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 422, in from_dict
    cache_info={
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 423, in <dictcomp>
    pkg_fmt: CachePackage.from_dict(info)
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 166, in from_dict
    return cls._class_per_type[d["_type"]](url=d["url"])
  File "/metaflow/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 127, in __init__
    for f in self.allowed_formats():
TypeError: 'NoneType' object is not iterable
I am currently trying to determine the best way to remote debug Python code that Metaflow runs in Argo in Kubernetes, but haven't gotten far yet.
d
Ah. I know whr that is. I am not going to be in front of a computer today but I can take a look this evening and fix. It’s due to the fact that you are not requesting a particular type of package (conda vs tarbz2) and so it is set to none but thst seems to be confusing the deserialization of thr environment. It should be a relatively easy fix.
c
Great. Thanks! I am not blocked. So, don't feel like you have to work the evening for me. :-)
I have 
"METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",
in 
/home/bbrandt/.metaflowconfig/config.json
when I create and trigger the workflow. Is this the variable it's looking for, but maybe I have to pass on the command-line instead?
Good news. If I applied the environment decorator with the variables needed for the extension in addition to having them set in the config.json, then I am able to get a successful run.
Copy code
@environment(vars={    
        "METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",
        "METAFLOW_CONDA_AZUREROOT":"preview-conda",
        "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba"
        })
It looks like for my simple experiment, runtime is about the same in Argo with and without the extension. The main value add I feel is the additional power for dependency management anyway. Here is the sample output from one step showing that bootstrapping environment is about 27 to 35 seconds per step on an Azure Standard_D4as_v5 node pool.
Copy code
Setting up task environment.
Downloading code package...
Code package downloaded.
Task is starting.
Bootstrapping environment ...
    Setting up Conda ... done in 0 seconds.
    Downloading 0(web) + 108(cache) packages ... done in 3 seconds.
    Extracting and linking Conda environment ... done in 32 seconds.
Environment bootstrapped.
alpha is 0.480000
I have not set any resource decorators, so I guess it using the defaults for CPU and memory to request from k8s. Argo lists this as the resource utilization for the workflow run: 
4m*(1 cpu),2m*(10Gi ephemeral-storage),1h*(100Mi memory)
Here is the code I am testing with:
Copy code
from metaflow import FlowSpec, Parameter, step, conda_base, environment

@conda_base(libraries={"pandas": "1.4.4", "python-duckdb": "0.6.1"})
class SimpleConda(FlowSpec):
    alpha = Parameter('alpha',
                      help='Learning rate',
                      default=0.01)

    @environment(vars={    
        "METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",
        "METAFLOW_CONDA_AZUREROOT":"preview-conda",
        "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba",
        "METAFLOW_DEBUG_CONDA":"1"
        })
    @step
    def start(self):
        print('alpha is %f' % self.alpha)
        
        self.next(self.pd_dates)

    @environment(vars={    
        "METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",
        "METAFLOW_CONDA_AZUREROOT":"preview-conda",
        "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba",
        "METAFLOW_DEBUG_CONDA":"1"
        })
    @step
    def pd_dates(self):
        import duckdb
        import pandas
        # connect to an in-memory database
        con = duckdb.connect()

        my_df = pandas.DataFrame.from_dict({'a': [42]})

        # query the Pandas DataFrame "my_df"
        self.results = con.execute("SELECT * FROM my_df").df()
        print(self.results)
        
        self.next(self.end)

    @environment(vars={    
        "METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",
        "METAFLOW_CONDA_AZUREROOT":"preview-conda",
        "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba",
        "METAFLOW_DEBUG_CONDA":"1"
        })
    @step
    def end(self):
        print('alpha is still %f' % self.alpha)

if __name__ == '__main__':
    SimpleConda()
I am running with:
Copy code
python param-ex/simple_conda.py  --environment=conda  --with retry argo-workflows create
python param-ex/simple_conda.py  --environment=conda  --with retry argo-workflows trigger --alpha 0.48
To test the 2 different configurations, I enable the extension with: 
pip install metaflow-netflixext --upgrade
I disable the extension with: 
pip uninstall metaflow-netflixext
For the performance numbers I am using a private AKS cluster, no noisy neighbors. Only running one job at a time, but doing multiple runs so I am not affected as much by things like node pool spin up/down.
d
Hey. Great. Yes, it is a limitation is OSs that you have to pass the env vars manually (I mention that in thr read me but will make it clearer as well as list this workaround. I’ll try on my machines with this setup to see if I am getting the same time. It looks like the majority of time is spent with the linking of thr env. it should be faster with .conda packages than tarbz2 packages (at least it was for me) so a bit surprising thst you are seeing no benefit compared to tarbz2. Whenever you get a chance if you could share the conda_v2.cnd file with me as well I can check thst nothing is amiss. You can also see METAFLOW_DEBUG_CONDA=1 to get a bit more detail (that may help me check it out). I’ll chedo this evening.
c
Maybe conda-lock would be faster than micromamba? Attached the 
METAFLOW_DEBUG_CONDA=1
output and the conda_v2.cnd from my local. Would there be a separate conda_v2.cnd I need to retrieve from container where the flow ran?
argo-azure-conda-debug.logconda_v2.cnd
d
And without the extension, it takes the same amount of time?
✅ 1
Based on thr log (cursory look), it looks like it is fetching and installing all packages as .conda from your cache.
I’ll take a closer look.
c
Is there something I messed up in the environment setup, maybe?
d
I’ll try it and see what I can find.
I’ll keep you posted shortly.
4 Views
Open in Slack
PreviousNext
Powered by