Hi all, I'm seeking for some help with the following setup. I have the following flow:

...
@step
def schedule_workers(self):
    worker_inputs = [...]
    self.next(self.run_worker, foreach='worker_inputs')

@catch(var='timeout')
@retry(times=0)
@timeout(seconds=3600)
@step
def run_worker(self):
    ...

...

Flows are deployed to aws step functions. I want the run to succeed in case run_worker tasks time out. However, the problem that I'm experiencing is that as soon as one of the run_worker tasks time out, other run_worker tasks are aborted, and the run fails (exception is not caught). Anyone has any suggestions how to fix this? Let me know if you need more explanation. Thanks!

Can you help us with the screenshots of both the jobs from the AWS Batch console?

hey @ancient-application-36103 , thanks for checking. is this enough, or do you need more info?

is this for the same run?

hey @ancient-application-36103, this is the same run, different tasks (which are run in parallel) which correspond to the

run_worker

step from my example above (in the screenshot the step is actually called

timed_step

and timeout is 1 min) one starts few mins after the other because the ec2 instance had to be provisioned in the background

Sorry for the delay. I've been looking into this and how to reproduce the issue. I have a hunch on the cause based on the AWS docs on how job timeouts are treated, but no luck reproducing it yet. Some further questions on the matter if you can help me out: • roughly how many tasks are in the foreach split? • are the processes in the

timed_step

such that they could possibly interfere with the internals of

@timeout

? specifically it relies on

signal

to break the process, so I'm wondering if a heavy process interferes with this enough that we do not send our own signal in time, instead falling to the AWS job timeout instead, which results in the exit 137

hey @bulky-afternoon-92433. thanks for looking into this. just fyi, I've found a different way to handle my use case, so don't feel like I'm rushing you or anything. the specific example I sent screenshots of was a short test sample (which I unfortunately deleted in the meantime). but basically the code was just running a for loop and sleeping (using

time.sleep

) I think there were only these two tasks in the foreach split however, the production code where I first noticed this issue had more tasks in split (around 10), and it did spin up another process in background

presumably your test flow was similar to this? And you were able to reproduce the exit 137 with just a sleep in the steps?

if I remember correctly, yes and yes

excellent, this should be enough info for me to work with. Also glad to hear that you were able to solve the use case in a different way 👌