Hi everyone I m running into this issue ```bash line 1 metaf Outerbounds #ask-metaflow

Hi everyone, I'm running into this issue. ```bash...

creamy-stone-99746

02/10/2023, 5:22 PM

Hi everyone, I'm running into this issue.

Copy code

bash: line 1: metaflow_VideoProcessingFlow_linux-64_042fc095966e314b3690f043716c87433a270f51/bin/python: No such file or directory

I've looked through the past threads, but I'm usin a completely fresh setup of metaflow and the services. I'm using

conda

NOT

mamba

✅ 1

ancient-application-36103

02/10/2023, 5:23 PM

are you using a custom docker image to run the task?

creamy-stone-99746

02/10/2023, 5:23 PM

No just this

Copy code

@batch(cpu=1, memory=8192)
    @conda(
        python="3.10.8",
        libraries={
            "conda-forge::av": "10.0.0",
            "conda-forge::Pillow": "9.4.0",
        },
    )

ancient-application-36103

02/10/2023, 5:24 PM

let me try to reproduce

creamy-stone-99746

02/10/2023, 5:25 PM

MF version

2.7.21

👍 1

creamy-stone-99746

02/10/2023, 5:26 PM

Thanks for the help. I have to step away for the next 30 mins

ancient-application-36103

02/10/2023, 5:27 PM

sure - @hallowed-glass-14538 do you have a few moments to take a look at this?

creamy-stone-99746

02/10/2023, 6:54 PM

Does it matter that I'm running the flow from a M1 mac and using batch? Is the conda env packaged on my mac and then sent to batch?

creamy-stone-99746

02/10/2023, 8:09 PM

Done some digging. I get this error

Copy code

bash: line 1: metaflow_HelloAWSFlow_linux-64_512c994a9e4deda56e1d4dcd3fce83fdeda634d3/bin/python: No such file or directory

I've confirmed that locally on my laptop the conda environment does contain the python executable.

Copy code

/opt/homebrew/anaconda3/envs/metaflow_HelloAWSFlow_linux-64_512c994a9e4deda56e1d4dcd3fce83fdeda634d3/bin/python

I can't run it locally (I'm on a mac) but it exists so it seems the issue is somewhere in this bash file.

creamy-stone-99746

02/10/2023, 8:21 PM

I've tried clearing the environments and having it build new ones.

creamy-stone-99746

02/10/2023, 8:21 PM

I've also rolled back to 2.7.10

creamy-stone-99746

02/10/2023, 8:21 PM

No luck same error.

hallowed-glass-14538

02/10/2023, 8:34 PM

Hey mike, were there any other logs from the container outside

Copy code

bash: line 1: metaflow_HelloAWSFlow_linux-64_512c994a9e4deda56e1d4dcd3fce83fdeda634d3/bin/python: No such file or directory

creamy-stone-99746

02/10/2023, 8:36 PM

Yes let me grab some

creamy-stone-99746

02/10/2023, 8:36 PM

It’s just boiler plate stuff I think

hallowed-glass-14538

02/10/2023, 8:36 PM

I was running with conda and I was unable to reproduce with metaflow 2.7.21 ;

Copy code

from metaflow import FlowSpec, step, conda, batch

class SimpleFlow(FlowSpec):

    @batch(cpu=1, memory=8192)
    @conda(
        python="3.10.8",
        libraries={
            "conda-forge::av": "10.0.0",
            "conda-forge::Pillow": "9.4.0",
        },
    )
    @step
    def start(self):
        print("Starting Flow")
        self.next(self.one)

    @step
    def one(self):
        self.next(self.join)
    
    @step
    def join(self):
        self.next(self.end)

    @step
    def end(self):
        print("Flow Completed")

if __name__ == "__main__":
    SimpleFlow()

hallowed-glass-14538

02/10/2023, 8:39 PM

My hunch is that it if the package is already on S3 then there may have been some download issue because of permissions.

creamy-stone-99746

02/10/2023, 8:40 PM

Copy code

Metaflow 2.7.22 executing HelloAWSFlow for user:mike
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping conda environment...(this could take a few minutes)
2023-02-10 12:38:28.183 Workflow starting (run-id 78):
2023-02-10 12:38:28.979 [78/start/338 (pid 55260)] Task is starting.
2023-02-10 12:38:31.841 [78/start/338 (pid 55260)] HelloAWS is starting.
2023-02-10 12:38:31.906 [78/start/338 (pid 55260)]
2023-02-10 12:38:31.906 [78/start/338 (pid 55260)] Using metadata provider: service@https://vg4zq92vg0.execute-api.us-west-1.amazonaws.com/api
2023-02-10 12:38:32.723 [78/start/338 (pid 55260)]
2023-02-10 12:38:32.723 [78/start/338 (pid 55260)] The start step is running locally. Next, the
2023-02-10 12:38:32.723 [78/start/338 (pid 55260)] 'hello' step will run remotely on AWS batch.
2023-02-10 12:38:32.723 [78/start/338 (pid 55260)] If you are running in the Netflix sandbox,
2023-02-10 12:38:32.723 [78/start/338 (pid 55260)] it may take some time to acquire a compute resource.
2023-02-10 12:38:32.962 [78/start/338 (pid 55260)] Task finished successfully.
2023-02-10 12:38:34.096 [78/hello/339 (pid 55298)] Task is starting.
2023-02-10 12:38:35.151 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Task is starting (status SUBMITTED)...
2023-02-10 12:38:36.045 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Task is starting (status RUNNABLE)...
2023-02-10 12:38:40.722 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Task is starting (status STARTING)...
2023-02-10 12:39:00.664 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Task is starting (status RUNNING)...
2023-02-10 12:38:56.392 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Setting up task environment.
2023-02-10 12:39:13.575 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Downloading code package...
2023-02-10 12:39:14.191 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Code package downloaded.
2023-02-10 12:39:14.213 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Task is starting.
2023-02-10 12:39:14.823 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Bootstrapping environment...
2023-02-10 12:39:37.062 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] Environment bootstrapped.
2023-02-10 12:39:37.062 [78/hello/339 (pid 55298)] [141cb4a5-a607-4771-961d-e6e3d1ebe019] bash: line 1: metaflow_HelloAWSFlow_linux-64_512c994a9e4deda56e1d4dcd3fce83fdeda634d3/bin/python: No such file or directory
2023-02-10 12:40:21.288 [78/hello/339 (pid 55298)] AWS Batch error:
2023-02-10 12:40:21.582 [78/hello/339 (pid 55298)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-02-10 12:40:21.582 [78/hello/339 (pid 55298)]
2023-02-10 12:40:21.984 [78/hello/339 (pid 55298)] Task failed.
2023-02-10 12:40:22.341 Workflow failed.
2023-02-10 12:40:22.341 Terminating 0 active tasks...
2023-02-10 12:40:22.341 Flushing logs...
    Step failure:
    Step hello (task-id 339) failed.

creamy-stone-99746

02/10/2023, 9:06 PM

@little-apartment-49355 I think that was it, I deleted the s3 object and now it's working.

curved-island-17262

02/11/2023, 2:00 AM

@creamy-stone-99746 I do face the same issue every now and then. I have to first delete the local conda env(

metaflow_VideoProcessingFlow_linux-64_042fc095966e314b3690f043716c87433a270f51

in your case) then remove the

.metaflow

directory to fix it.

curved-island-17262

04/12/2023, 7:40 AM

@brief-kite-90012

👀 1

astonishing-father-21369

12/06/2023, 6:33 AM

hello, I am running into this now using conda and batch and wondering how to find the S3 objects I may need to remove? the flow I am struggling with runs fine locally without batch. thanks very much.

ancient-application-36103

12/07/2023, 3:45 PM

@astonishing-father-21369 what version of metaflow are you on?

astonishing-father-21369

12/08/2023, 5:10 AM

@square-wire-39606 2.8.5

Open in Slack

Previous Next