Hello, I have been using metaflow with AWS batch f...
# ask-metaflowa
astonishing-father-21369
12/13/2023, 1:10 PMHello, I have been using metaflow with AWS batch for over a year without problems from my apple laptop. I recently seem to have mismanaged my local conda environments such that when I run with batch now, no flows that use conda seem able to work. more details in thread
astonishing-father-21369
12/13/2023, 1:11 PMI am now seeing errors of this sort which I know have popped up elsewhere:
Copy code
(metaflow-jobs) ➜ dl-metaflow-jobs git:(main) ✗ python helloworld.py --environment=conda --with batch run
Metaflow 2.9.7 executing HelloFlow for user:mcooper
Validating your flow...
The graph looks good!
Running pylint...
Pylint is happy!
Bootstrapping conda environment...(this could take a few minutes)
2023-12-12 09:45:30.355 Workflow starting (run-id 107285):
2023-12-12 09:45:32.555 [107285/start/1288183 (pid 53752)] Task is starting.
2023-12-12 09:45:34.621 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status SUBMITTED)...
2023-12-12 09:45:40.243 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status RUNNABLE)...
2023-12-12 09:46:10.293 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status RUNNABLE)...
2023-12-12 09:46:40.323 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status RUNNABLE)...
2023-12-12 09:47:10.473 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status RUNNABLE)...
2023-12-12 09:47:17.222 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status STARTING)...
2023-12-12 09:47:36.905 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting (status RUNNING)...
2023-12-12 09:47:35.859 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Setting up task environment.
2023-12-12 09:47:44.598 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Downloading code package...
2023-12-12 09:47:45.288 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Code package downloaded.
2023-12-12 09:47:45.315 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Task is starting.
2023-12-12 09:47:46.324 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Bootstrapping environment...
2023-12-12 09:48:09.141 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] Environment bootstrapped.
2023-12-12 09:48:09.141 [107285/start/1288183 (pid 53752)] [f3148b21-61b5-4604-a2b8-c4a2ace9da0c] bash: line 1: metaflow_HelloFlow_linux-64_14f5715f8a113e26a97d33572cf52d8ed1b0819c/bin/python: No such file or directory
2023-12-12 09:48:13.046 [107285/start/1288183 (pid 53752)] AWS Batch error:
2023-12-12 09:48:13.046 [107285/start/1288183 (pid 53752)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-12-12 09:48:13.128 [107285/start/1288183 (pid 53752)]
2023-12-12 09:48:13.727 [107285/start/1288183 (pid 53752)] Task failed.
2023-12-12 09:48:14.128 Workflow failed.
2023-12-12 09:48:14.128 Terminating 0 active tasks...
2023-12-12 09:48:14.129 Flushing logs...
Step failure:
Step start (task-id 1288183) failed.astonishing-father-21369
12/13/2023, 1:13 PMalmost all flows that I have tested that fail with batch complete successfully when running locally. Is there something in S3 I need to regenerate? I have tried with other flows to change libraries specified using conda_base to see if the batch environment rebuilds, but nothing has helped yet.
8 Views