Hey all I am trying to create an isolated environment for my Outerbounds #ask-metaflow

Hey all, I am trying to create an isolated environ...

curved-market-11906

04/02/2024, 4:51 PM

Hey all, I am trying to create an isolated environment for my flow using the

@conda

and

@pypi

decorators. One of the libraries I need for my flow (

metaflow-card-html

) is only available in

pypi

(all other libraries are available via

conda

) and therefore I need to combine both decorators. I installed the

metaflow-netflixext

in order to be able to use both

@conda

and

@pypi

decorators in the same flow for the time being. However when running my flow on

AWS

batch I run into a

Micromamba

error:

Copy code

Stdout: Failed to install Micromamba!

I tried uninstalling

metaflow-netflixext

and running only on the stable Metaflow version, and executing a command to manually pip install

metaflow-card-html

before the FlowSpec, like so:

Copy code

import subprocess
import sys

subprocess.run([sys.executable, "-m", "pip", "install", "metaflow-card-html"])

and then ran the flow with

python path/to/flow.py --environment=conda run

but ran into the same Micromamba error. Wondering if anyone has seen this

Micromamba

error before and knows a solution?

✅ 1

curved-market-11906

04/02/2024, 4:52 PM

full error message:

Copy code

Bootstrapping virtual environment...
2024-04-02 16:12:14.303 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] Bootstrap failed while executing: if ! command -v ./micromamba >/dev/null 2>&1; then
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]             wget -qO- <https://micro.mamba.pm/api/micromamba/linux-64/latest> | python -c "import sys, bz2; sys.stdout.buffer.write(bz2.decompress(sys.stdin.buffer.read()))" | tar -xv bin/micromamba --strip-components=1 ;
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]             if ! command -v ./micromamba >/dev/null 2>&1; then
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]                 echo "Failed to install Micromamba!";
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]                 exit 1;
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]             fi;
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]         fi
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] Stdout: Failed to install Micromamba!
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6]
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] Stderr: /bin/sh: 2: wget: not found
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] tar: This does not look like a tar archive
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] tar: bin/micromamba: Not found in archive
2024-04-02 16:12:14.543 [229400/start/2739208 (pid 2273369)] [6cc331a5-75a7-46d3-961c-a9d8faebf8a6] tar: Exiting with failure status due to previous errors

dry-beach-38304

04/02/2024, 4:53 PM

looking.

dry-beach-38304

04/02/2024, 6:07 PM

(haven’t forgotten, dealing with anothe rissue at work)

curved-market-11906

04/02/2024, 6:08 PM

all good - appreciate you taking a 👀

dry-beach-38304

04/02/2024, 6:55 PM

ah, it’s because there is no wget on your box.

dry-beach-38304

04/02/2024, 6:55 PM

I tested the code and there doesn’t seem to be a change in the micromamba executable.

dry-beach-38304

04/02/2024, 6:56 PM

although wait, why it is looking for wget

dry-beach-38304

04/02/2024, 6:56 PM

this looks like the regular pypi decorator. Let me confirm.

dry-beach-38304

04/02/2024, 7:00 PM

You may be using an old version of metaflow and also not the netflix extension (or something went wrong). The line being referenced in your error message was replaced in OSS (non extension) in Oct 2023: https://github.com/Netflix/metaflow/pull/1601/files

dry-beach-38304

04/02/2024, 7:02 PM

So my recommendation would be to either update metaflow or install wget on your remote box. The extension should work regardless (as in, you should not get “Failed to install Micromamba!” with the extension so something else may be going on there). I would suggest a clean install and retrying.

elegant-lamp-40213

04/02/2024, 8:09 PM

Have you thought about creating the relevant conda package on

conda-forge

? This way you may be able to leverage only the @conda decorator and work around the issue?

curved-market-11906

04/03/2024, 2:52 AM

Yeah okay I am currently running on metaflow

2.10.0

- it looks like this change went into effect as of

2.10.14

, and after testing my flow on metaflow version

2.10.14

it ran successfully.

26 Views

Open in Slack

Previous Next