https://outerbounds.com/ logo
Join Slack
Powered by
Hi team, I am trying to use devstack to run one of...
# ask-metaflow
w
Hi team, I am trying to use devstack to run one of my flow using kubernetes but getting the following error, wondering if I can get some help with this
Copy code
poetry run python metaflow_jobs/flows/train.py --with kubernetes --environment=pypi run
Metaflow 2.19.7 executing CovtypeTrainingPipelineFlow for user:chen.liang
Project: covtype, Branch: user.chen.liang
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2025-11-14 08:51:19.058 Bootstrapping virtual environment(s) ...
2025-11-14 08:51:19.213 Virtual environment(s) bootstrapped!
2025-11-14 08:51:19.577 Workflow starting (run-id 2), see it in the UI at <http://localhost:3000/CovtypeTrainingPipelineFlow/2>
2025-11-14 08:51:20.588 [2/start/4 (pid 81134)] Task is starting.
2025-11-14 08:51:21.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2025-11-14 08:51:21.689 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Setting up task environment.
2025-11-14 08:51:26.033 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Downloading code package...
2025-11-14 08:51:26.498 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Code package downloaded.
2025-11-14 08:51:26.526 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Task is starting.
2025-11-14 08:51:26.913 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Bootstrapping virtual environment...
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Bootstrap failed while executing: set -e;
2025-11-14 08:51:34.444 [2/start/4 (pid 81134)] Kubernetes error:
2025-11-14 08:51:34.444 [2/start/4 (pid 81134)] Error: Setting up task environment.
2025-11-14 08:51:34.541 [2/start/4 (pid 81134)] Downloading code package...
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Code package downloaded.
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Task is starting.
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Bootstrapping virtual environment...
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Bootstrap failed while executing: set -e;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] tmpfile=$(mktemp);
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] echo "@EXPLICIT" > "$tmpfile";
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] ls -d /metaflow/.pkgs/conda// >> "$tmpfile";
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export PATH=$PATH:$(pwd)/micromamba;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export CONDA_PKGS_DIRS=$(pwd)/micromamba/pkgs;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export MAMBA_NO_LOW_SPEED_LIMIT=1;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export MAMBA_USE_INDEX_CACHE=1;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export MAMBA_NO_PROGRESS_BARS=1;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] export CONDA_FETCH_THREADS=1;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] micromamba create --yes --offline --no-deps                 --safety-checks=disabled --no-extra-safety-checks                 --prefix /metaflow/linux-aarch64/538301e1d2a9acc --file "$tmpfile"                 --no-pyc --no-rc --always-copy;
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] rm "$tmpfile"
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Stdout:
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] Stderr: ls: cannot access '/metaflow/.pkgs/conda//': No such file or directory
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] 
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] (exit code 1). This could be a transient error. Use @retry to retry.
2025-11-14 08:51:34.542 [2/start/4 (pid 81134)] 
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             tmpfile=$(mktemp);
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             echo "@EXPLICIT" > "$tmpfile";
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             ls -d /metaflow/.pkgs/conda/*/* >> "$tmpfile";
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export PATH=$PATH:$(pwd)/micromamba;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export CONDA_PKGS_DIRS=$(pwd)/micromamba/pkgs;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export MAMBA_NO_LOW_SPEED_LIMIT=1;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export MAMBA_USE_INDEX_CACHE=1;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export MAMBA_NO_PROGRESS_BARS=1;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             export CONDA_FETCH_THREADS=1;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             micromamba create --yes --offline --no-deps                 --safety-checks=disabled --no-extra-safety-checks                 --prefix /metaflow/linux-aarch64/538301e1d2a9acc --file "$tmpfile"                 --no-pyc --no-rc --always-copy;
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]             rm "$tmpfile"
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Stdout:
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29] Stderr: ls: cannot access '/metaflow/.pkgs/conda/*/*': No such file or directory
2025-11-14 08:51:33.352 [2/start/4 (pid 81134)] [pod t-07ea3803-xkdgx-jth29]
2025-11-14 08:51:34.553 [2/start/4 (pid 81134)] Task failed.
2025-11-14 08:51:34.562 Workflow failed.
2025-11-14 08:51:34.562 Terminating 0 active tasks...
2025-11-14 08:51:34.562 Flushing logs...
    Step failure:
    Step start (task-id 4) failed.
My devstack is setup through this command and looks like everything goes fine in tilt
Copy code
MINIKUBE_MEMORY=max MINIKUBE_CPUS=max metaflow-dev all-up
Inside the code I was doing these at flow level, nothing fancy
Copy code
@project(name="covtype")
@pypi_base(
    packages={
        "pandas": "2.0.0",
        "numpy": "1.26.4",
        "scikit-learn": "1.7.2",
        "torch": "2.4.1",
        "pytorch-lightning": "2.5.0.post0",
        "matplotlib": "3.10.5",
    }
)
8 Views
Open in Slack
PreviousNext
Powered by