Hey! I'm attempting to use the

pypi

decorator for package management, but when I run this in the

metaflow-dev

stack, I get an error saying that Metaflow can't find Conda artifacts. Has anyone seen this before, and is it expected? This is my invocation + output:

❯ python ubuntu_updates.py --environment=pypi run --with kubernetes                                                                                                                                                                                   [2/5008]
Metaflow 2.15.10 executing UbuntuUpdatesFlow for user:coder
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2025-05-05 21:50:24.808 Bootstrapping virtual environment(s) ...
2025-05-05 21:50:24.881 Virtual environment(s) bootstrapped!
2025-05-05 21:50:25.263 Workflow starting (run-id 4), see it in the UI at <http://localhost:3000/UbuntuUpdatesFlow/4>
2025-05-05 21:50:25.673 [4/start/8 (pid 434281)] Task is starting.
2025-05-05 21:50:26.493 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2025-05-05 21:50:27.218 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Setting up task environment.
2025-05-05 21:50:32.327 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Downloading code package...
2025-05-05 21:50:32.938 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Code package downloaded.
2025-05-05 21:50:32.977 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Task is starting.
2025-05-05 21:50:33.839 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Bootstrapping virtual environment...
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Bootstrap failed while executing: set -e;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             tmpfile=$(mktemp);
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             echo "@EXPLICIT" > "$tmpfile";
2025-05-05 21:50:37.212 [4/start/8 (pid 434281)] Kubernetes error:
2025-05-05 21:50:37.212 [4/start/8 (pid 434281)] Error: Setting up task environment.
2025-05-05 21:50:37.212 [4/start/8 (pid 434281)] Downloading code package...
2025-05-05 21:50:37.212 [4/start/8 (pid 434281)] Code package downloaded.
2025-05-05 21:50:37.212 [4/start/8 (pid 434281)] Task is starting.
2025-05-05 21:50:37.213 [4/start/8 (pid 434281)] Bootstrapping virtual environment...
2025-05-05 21:50:37.213 [4/start/8 (pid 434281)] Bootstrap failed while executing: set -e;
2025-05-05 21:50:37.213 [4/start/8 (pid 434281)] tmpfile=$(mktemp);
2025-05-05 21:50:37.213 [4/start/8 (pid 434281)] echo "@EXPLICIT" > "$tmpfile";
2025-05-05 21:50:37.213 [4/start/8 (pid 434281)] ls -d /metaflow/.pkgs/conda// >> "$tmpfile";
2025-05-05 21:50:37.345 [4/start/8 (pid 434281)] export PATH=$PATH:$(pwd)/micromamba;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] export CONDA_PKGS_DIRS=$(pwd)/micromamba/pkgs;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] export MAMBA_NO_LOW_SPEED_LIMIT=1;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] export MAMBA_USE_INDEX_CACHE=1;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] export MAMBA_NO_PROGRESS_BARS=1;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] export CONDA_FETCH_THREADS=1;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] micromamba create --yes --offline --no-deps                 --safety-checks=disabled --no-extra-safety-checks                 --prefix /metaflow/linux-64/f35ade658f6977a --file "$tmpfile"                 -
-no-pyc --no-rc --always-copy;
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] rm "$tmpfile"
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] Stdout:
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] Stderr: ls: cannot access '/metaflow/.pkgs/conda//': No such file or directory
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)]
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)] (exit code 1). This could be a transient error. Use @retry to retry.
2025-05-05 21:50:37.346 [4/start/8 (pid 434281)]
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             ls -d /metaflow/.pkgs/conda/*/* >> "$tmpfile";
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export PATH=$PATH:$(pwd)/micromamba;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export CONDA_PKGS_DIRS=$(pwd)/micromamba/pkgs;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export MAMBA_NO_LOW_SPEED_LIMIT=1;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export MAMBA_USE_INDEX_CACHE=1;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export MAMBA_NO_PROGRESS_BARS=1;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             export CONDA_FETCH_THREADS=1;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             micromamba create --yes --offline --no-deps                 --safety-checks=disabled --no-extra-safety-checks                 --prefix /metaflow/linux-64/f35ade658f
6977a --file "$tmpfile"                 --no-pyc --no-rc --always-copy;
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]             rm "$tmpfile"
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Stdout:
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v] Stderr: ls: cannot access '/metaflow/.pkgs/conda/*/*': No such file or directory
2025-05-05 21:50:35.659 [4/start/8 (pid 434281)] [pod t-840c2468-jfv52-wt56v]
2025-05-05 21:50:37.362 [4/start/8 (pid 434281)] Task failed.
2025-05-05 21:50:37.381 Workflow failed.
2025-05-05 21:50:37.382 Terminating 0 active tasks...
2025-05-05 21:50:37.382 Flushing logs...
    Step failure:
    Step start (task-id 8) failed.

Any info would be useful, thanks!!

did you figure this out?

Hey same here, did you figure this out?

We didn't figure this out unfortunately. We worked around it by using custom images