I’m getting occasional failures when running on k8s that return “exit code None” at which point it doesn’t retry the job. Are there rules somewhere in the code about when a retry is attempted? I have the retry decorator on the task and it does successfully retry other errors for the same task (e.g. OOM)

that's interesting - can you help us with the stack trace?

there’s no stack trace, nothing in stderr in the logs, and nothing printed to the cli other than the exit code:

2023-04-12 14:05:46.257 [337/eval/1903 (pid 876397)] [pod t-td4jn-w4gmp] Task finished with exit code None.
2023-04-12 14:05:46.811 Workflow failed.                                                                        
2023-04-12 14:05:46.811 Terminating 4 active tasks...

can you speak a bit more about your setup? are you using any custom decorators?

I can speak some more to this. We've deployed on kubernetes using the metaflow AWS module Here's the decorators for that particular function:

@kubernetes(cpu=2, memory=72000, namespace="argo")
    @retry
    @step
    def eval(self):

It's odd in that the failures are intermittent We are using spot instances, so my current theory is that when the EC2 instance is occasionally terminated due to spot instance interruption, Metaflow isn't given a clean exit code to go and retry the step. I've manually tested this theory by starting a job and then terminating the EC2 instance that the node is scheduled on. Here was my code and resulting error:

from metaflow import FlowSpec, step, kubernetes, retry

class ForeachFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.a)
        
        
    @kubernetes(cpu=2, memory=74000)
    @retry
    @step
    def a(self):
    
        import time
        for i in range(720):
            print(i)
            time.sleep(1)
        self.next(self.end)

    @step
    def end(self):
        print('done')

if __name__ == '__main__':
    ForeachFlow()

2023-04-12 15:39:47.981 [341/a/1977 (pid 19866)] [pod t-rzz57-9rps9] 80
2023-04-12 15:39:48.982 [341/a/1977 (pid 19866)] [pod t-rzz57-9rps9] 81
2023-04-12 15:40:33.425 [341/a/1977 (pid 19866)] [pod t-rzz57-9rps9] Task finished with exit code None.
2023-04-12 15:40:34.113 Workflow failed.
2023-04-12 15:40:34.113 Terminating 1 active tasks...
2023-04-12 15:40:34.114 [341/a/1977 (pid 19866)] [KILLED BY ORCHESTRATOR]
2023-04-12 15:40:34.114 [341/a/1977 (pid 19866)] [KILLED BY ORCHESTRATOR]
2023-04-12 15:40:34.114 Flushing logs...
    Data store error:
    No completed attempts of the task was found for task 'ForeachFlow/341/a/1977'

thanks! this helps us debug this issue better. i will follow up as soon as I have details to share

So I have some more info on this. It looks like our autoscaler is triggering pods for scale-down even though there are running metaflow jobs on them. This is after we switched to ondemand node types. Is there a way to add a kubernetes annotation to all the metaflow jobs so the autoscaler knows not to kill them? The following is what I'm trying to add:

"<http://cluster-autoscaler.kubernetes.io/scale-down-disabled|cluster-autoscaler.kubernetes.io/scale-down-disabled>": "true"

@elegant-beach-10818 can you help me with more details on your set up? I haven't been able to replicate this specific issue on my end.

@square-wire-39606 Sure! Sometime this afternoon work for you? I'm available from 11AM-5PM CT

would you be around sometime tomorrow? cc @User

tomorrow works

invite sent!