Hello, I was wondering if anyone has suggestions for isolating branched steps using the @catch decorator. Currently I am doing some data processing in a step, and this data processing can be problematic and cause various issues. To prevent a failure in this step from causing the entire flow to fail I am using a @catch and @timeout to limit the allowed execution time as well as catch exceptions, and I am also executing the processing in a subprocess as well to handle segmentation faults. The issue I'm running into is that when a memory leak occurs, which sometimes happens, the step hard fails (seems like kubernetes is killing the pod), causing the entire flow to eventually fail and not complete. Should the I am using Argo to start my Metaflows. I can provide more detail if it helps. Would using a custom step decorator monitoring memory usage be a good way to go? Thanks!

If you have the @catch decorator - then all failures are caught for that step - including segfaults and ooms - is that not what you are observing?

Correct, seg faults were not being caught, I used subprocess.open to execute the processing in a subprocess and watch for segfaults, that worked. For OOM's the catch decorator isn't picking it up either. The way I notice the failure is in looking in my k8s cluster and seeing the OOMKiller killed that pod which prevents me from managing that failure through Metaflow.

Do you have a reproducible example for us? Perhaps one that works on dev stack?

I don't. I'll try some experiments to see if I can manually reproduce it. It doesn't even happen for the same file twice, it seems random. I'll try to see if I can come way to demo it.