Hello! I'm facing issues with environment resolution when trying to run a flow where it gets stuck on Resolving the environment. I set the conda debugging on and see that it is stuck at this :

debug[conda /local/home/unmeshph/.cache/pypoetry/virtualenvs/saitama-SKj-zWo--py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:317]: Environment is of type pip-only
debug[conda /local/home/unmeshph/.cache/pypoetry/virtualenvs/saitama-SKj-zWo--py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1984]: Creating builder conda environment

Any pointers on debugging this further?

this means that it is trying to create a conda environment. Can you see what command it is running (you can use ps and check for anything running conda or mamba or micromamba)

Do you know what is the directory containing the cache?

it depends on your installation. What is the commandline being executed? And then you can use the conda/mamba/micromamba binary to run <bin> info and it should tell you.

So my main command is running a flow with the

--environment=conda

flag. I cleared the caches for miniforge but same result. I think I need to clean up some other caches too ?

sorry, I meant the command it is running to create the builder environment

Here's the debug log

oh — this is a very old version of the extensions. Could you update to 1.1.2 please and retry?

okay I updated both. Here's the new one:

ok, so yes, it is basically doing this:

/home/unmeshph/miniforge3/bin/mamba create --prefix /tmp/tmpp22df080/prefix --dry-run 'python==3.10.12' pip wheel tomli setuptools

and that is getting stuck.

I ran it now, looks like it asks for a user prompt:

Continue creating environment (y/[n])? y

is that so? OK, Well that would explain it. What version of mamba do you have. Mine doesn’t do that

yeah that seems to work

ok, so which version of mamba for my own education.

yes can wait till tonight. Version of mamba is 1.4.2

intersting. that’s an older version. may be some flags in your config or something. It’s really weird that i have never seen this behavior.

should I try and upgrade mamba?

I don’t know. I’m surprised it doesn’t work for that one. I am pretty sure I had used it in the past. I would check if you have anything in your config files. I can’t find any setting that seems to have an impact (tried always_yes, extra_safety_checks)

hmm okay will check, thanks for the help on this. If it still doesn't work, will wait for the fix

One line change: https://github.com/Netflix/metaflow-nflx-extensions/pull/33. Could you try that change locally and let me know. If all good, I’ll go ahead and merge. I did have it in another place but not here for some weird reason.

Hey @dry-beach-38304, looks like it is still getting stuck at the same place

did you add the “Priting arguments to the conda call”. Does it work in other cases. If so, can you send me the simple flow you are trying to do so I can try to reproduce it

Hey Romain, sorry for the late response. We digged around a bit more and found that there were some stray environment creation lock files lying around in the mamba directory which was causing the env resolution to get stuck. I believe these lock files were created since I had interrupted previous environment resolution attempts. Maybe this is something that can be accounted by having some cleanup to remove these locks?

Hey. I do create lock files but it should print out a message if thr lock isnt acquired within some time. I’ll check if that logic is broken. In other words, when it tries to acquire a lock, if it can’t, after some time it lets the user know. At least for the locks I create. Which lock ended up being the one that solved the problem?

There was a lock in the mamba directory which was called something like

mf-env.lock

hun. That’s weird. I can’t find that one in the code anywhere (I have mf_env-creation.lock mf_pkgs-update.lock. Maybe from some older version. At any rate, it all works now?

Ohh yeah I think those are the two. I did not remember the name exactly

Ok — they should print a message if stuck for a while. I’ll check if all that is correct.

Yeah I have had runs where it prints the message, but it wasn't doing so for the env resolution this time. At any rate, deleting those locks resolved it