Been having issues with tasks failing without any ...
# ask-metaflow
b
Been having issues with tasks failing without any meaningful error message. It just says
Task failed.
I've been debugging it for a while and all the business logic defined in the task seems to be running successfully and then it almost seems like the task fails as it's exiting. And thus the whole workflow stops Extra context: • I'm running metaflow purely locally right now (i.e. no
.metaflowconfig
) • I'm saving a list of instances of a custom class at the end of that step • I'm using the
foreach
statement and some other parallel tasks of the same step seem to be exiting / going to the next step fine Does this sound familiar to anyone at all?
1
b
Are you seeing
Task failed
in the UI, or the command line?
b
In the command line. Some more context: • I had 100 objects to process, so I chunked them into groups of 10 and used
foreach
to process them in parallel, and when I did that some of the tasks would fail • So I kept the 100 objects and tried processing them with a single task (i.e. not in parallel) and it would get to the very end and fail at the end • I just tried chunking them again into groups of 10, but set
max-workers
to 1 and it just finished successfully Trying that second bullet again right now to confirm that that is actually what happens. If that one succeeds, then I'm ok with chalking it up to some issue with a bunch of processes running in parallel on my laptop
yeah it failed again with a single step to process all 100 objects. failed at the very end after everything had finished processing 🤷‍♂️