< WM13WM6JC> Since I wiped and redeployed my AKS cluster thi Outerbounds #ask-metaflow

<@WM13WM6JC> Since I wiped and redeployed my AKS c...

cuddly-family-48123

02/28/2023, 4:57 AM

@User Since I wiped and redeployed my AKS cluster this is the error I am seeing with the @conda decorator when run through Argo in the current metaflow release. As I had mentioned in our chat, it works great as soon as I

pip install metaflow-netflixext --upgrade

to use the v2 @conda decorator.

Copy code

Setting up task environment.
Downloading code package...
Code package downloaded.
Task is starting.
Bootstrapping environment...
Environment bootstrapped.
/bin/bash: line 1: metaflow_MinimumFlow_linux-64_c5b6dc01d4f4a08e54fef899876a247da78dfef5/bin/python: No such file or directory

Running:

Copy code

python minimum_flow.py --environment=conda --with retry argo-workflows create
python minimum_flow.py --environment=conda --with retry argo-workflows trigger

Pretty simple:

Copy code

from metaflow import FlowSpec, step, project, conda_base, environment

@project(name='minflow')
@conda_base(python="3.10.9",libraries={"pandas": "1.4.4", "python-duckdb": "0.6.1"})
class MinimumFlow(FlowSpec):
       
    @step
    def start(self):
        self.next(self.end)

    @step
    def end(self):
        print("Flow is done!")

if __name__ == "__main__":
    MinimumFlow()

dry-beach-38304

02/28/2023, 10:04 AM

hum, did you check if there is a

conda.cnd

file in a

.metaflow

directory somewhere up your path (so either in the directory you run from or its parent, or its parent etc). I wonder if some of the cached information stuck around and conda v1 got confused. This seems to indicate that it didn’t create the environment properly.

cuddly-family-48123

02/28/2023, 3:22 PM

No. I didn't see anything in the docs about conda.cnd. I do not see a conda.cnd anywhere in the directory tree. I do see these files: • .metaflow/conda_v2.cnd • .metaflow/MinimumFlow/conda.dependencies

conda_v2.cnd conda.dependencies

cuddly-family-48123

02/28/2023, 3:27 PM

This is how I removed the conda V2 extension to test the old conda implementation:

Copy code

bbrandt@939TQ13:~/src/tutorials$ pip uninstall metaflow-netflixext
Found existing installation: metaflow-netflixext 0.0.7
Uninstalling metaflow-netflixext-0.0.7:
  Would remove:
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/cmd/environment/*
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/cmd/mfextinit_netflixext.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/config/mfextinit_netflixext.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/*
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/environment_cli.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/mfextinit_netflixext.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/toplevel/mfextinit_netflixext.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/toplevel/netflixext_toplevel.py
    /home/bbrandt/mambaforge/envs/intro-to-mf/lib/python3.10/site-packages/metaflow_netflixext-0.0.7.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled metaflow-netflixext-0.0.7

cuddly-family-48123

02/28/2023, 4:00 PM

The versions of argo and metaflow I have installed in my AKS cluster as visible in this PR: https://github.com/outerbounds/metaflow-tools/pull/33/files#diff-7e625468b65e454b156177b6e2295e4aa85f377b96e51e9a378d09c28c75e05bR22 My local argo version is:

Copy code

~/src/tutorials$ argo version
argo: v3.4.5
  BuildDate: 2023-02-07T13:22:11Z
  GitCommit: 1253f443baa8ad1610d2e62ec26ecdc85fe1b837
  GitTreeState: clean
  GitTag: v3.4.5
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

dry-beach-38304

02/28/2023, 4:42 PM

ah sorry, in oss it is the conda.dependencies file. Try trashing that one fully and see if that helps with the original conda.

cuddly-family-48123

02/28/2023, 5:30 PM

I deleted the conda.dependencies and reran the argo create and trigger. I get the same error from the start/main. Attaching the log from the start/wait that logs some of the setup.

azure-argo-conda-failure-start-wait.log

cuddly-family-48123

02/28/2023, 5:32 PM

I'm happy to help you track it down. Let me know what info you'd need. Once this starts working I can provide you with the timings to compare v1 and v2 conda implementation you requested. I am not otherwise blocked for any other work I need to do though if you need to save it for another time.

cuddly-family-48123

02/28/2023, 5:38 PM

These are the lines from that log that seemed strange to me:

Copy code

time="2023-02-28T17:16:05.563Z" level=error msg="executor error: open /mainctrfs/mnt/out/task_id: no such file or directory"
time="2023-02-28T17:16:05.663Z" level=fatal msg="open /mainctrfs/mnt/out/task_id: no such file or directory"

ancient-application-36103

02/28/2023, 11:25 PM

@cuddly-family-48123 can you help me with the output of

python flow.py argo-workflows create --only-json

cuddly-family-48123

02/28/2023, 11:57 PM

Wow. Didn't know you could do that. Here's the command and output @ancient-application-36103.

Copy code

python minimum_flow.py --environment=conda --with retry argo-workflows create --only-json > minimum_flow-argo-create.json

minimum_flow-argo-create.json

2 Views

Open in Slack

Previous Next