bright-author-73055
09/25/2023, 9:49 PMTask failed. I've been debugging it for a while and all the business logic defined in the task seems to be running successfully and then it almost seems like the task fails as it's exiting. And thus the whole workflow stops
Extra context:
• I'm running metaflow purely locally right now (i.e. no .metaflowconfig)
• I'm saving a list of instances of a custom class at the end of that step
• I'm using the foreach statement and some other parallel tasks of the same step seem to be exiting / going to the next step fine
Does this sound familiar to anyone at all?brave-lion-15961
09/25/2023, 10:06 PMTask failed in the UI, or the command line?bright-author-73055
09/25/2023, 10:10 PMforeach to process them in parallel, and when I did that some of the tasks would fail
• So I kept the 100 objects and tried processing them with a single task (i.e. not in parallel) and it would get to the very end and fail at the end
• I just tried chunking them again into groups of 10, but set max-workers to 1 and it just finished successfully
Trying that second bullet again right now to confirm that that is actually what happens. If that one succeeds, then I'm ok with chalking it up to some issue with a bunch of processes running in parallel on my laptopbright-author-73055
09/25/2023, 10:21 PM