https://outerbounds.com/ logo
Join Slack
Powered by
Hey all. My batch jobs are failing with: `Environ...
# ask-metaflow
s
Hey all. My batch jobs are failing with: `Environment bootstrapped bash: line 1: metaflow_[
…
]/bin/python: no such file or directory` I’m using retry decorators and that’s not working either. Current setup: python3.10 (using that same docker image too — not a custom one). Using conda_base decorator. Adding a subdirectory to my Python path, in case you use that somewhere Any help would be appreciated
I’m trying to use this at work but i haven’t been able to get it to work yet at all
a
are you setting any memory resources for this step?
s
I am. 32Bg
Gb *
It seems strange that this would be a memory issue to me
a
conda needs some memory to bootstrap - if you specify 1G, then it will definitely run into issues. 32G should be more than enough
Are you using any custom extensions?
s
Ah I see
No I didn’t even know there were any to speak of
a
to make sure that this isn't an issue with a corrupt conda package, can you run a flow without the 
@conda_base
decorator but still with
Copy code
--environment=conda run --with batch
s
So I should have the requirements in my active conda env, right?
a
if you are using the conda integration, then your current active environment has no bearing to the environment that's created for executing metaflow tasks
s
Ah, so this is to see if it spins up at all. Gotcha
Let me see
But btw, the first few jobs in this foreach succeed
After 30 or 40 of them, then they start to fail
Yup. I’m getting no module named…
I tried doing the same with 
resume
and it looks the same
a
yeah you would get the 
no module named
error if you have an import for which the package doesn't exist
are all these jobs running on AWS Batch?
s
Not all the jobs in the dag, but most. And obviously all of the jobs for this step in the foreach
a
but some succeed and yet other fail?
for the foreach tasks?
s
Yessir
a
what's the error trace?
s
It doesn’t even get to my code. So it’s literally just the “environment bootstrapped” and then “…/Python: no such file or directory”
It seems like the artifacts aren’t finished downloading (or extracting if they’re compressed)
a
The newest execution attempt is also showing the “…/Python: no such file or directory” error?
s
Ah no sorry. Those are:
Copy code
Internal error
Traceback (most recent call last):
File “/metaflow/metaflow/task.py”, line 586, in run_step
File “/metaflow/metaflow/task.py”, line 60, in _exec_step_function
Step_function()
… 
No module named pandas
a
that's expected because you are importing pandas somewhere in the code and we removed that dependency
there is a local 
.metaflow
folder. can you nuke that and reexecute your original flow on AWS Batch?
👍 1
s
I’m still importing pandas right?
Or should I change my code so it just sleeps or something?
a
that's fine - even if it fails with this error. the 
…/Python: no such file or directory
error happens before the 
pandas
error.
s
Well now it’s not even getting to batch
a
what's the error?
s
(I use pandas before batch)
No pandas again
a
@batch is a decorator - I didn't get how you use pandas before batch
s
It’s going through click and then metaflow task.
Oh I have local tasks too (because they’re faster to spin up)
So I use pandas in these local tasks
a
you could run a dummy flow that has the same 
@conda_base
decorator as your original flow
s
Would it help if I delete the conda envs and start over?
a
do you have 
pandas
specified as a dependency in 
@conda_base
for this execution?
sure
s
And yes I had it as a dependency in conda_base, but then I commented out the conda_base decorator like you asked
a
can you uncomment it, nuke the 
.metaflow
folder and reexecute?
s
Doing that right now
I’ve also removed those envs
a
Did it work?
7 Views
Open in Slack
PreviousNext
Powered by