Hola we are getting this errors when running our flows in k8 Outerbounds #ask-metaflow

Hola, we are getting this errors, when running our...

fast-pizza-24629

08/29/2023, 4:55 PM

Hola, we are getting this errors, when running our flows in k8s... and we haven't been able to figure out why... I was looking for something similar here but they were related to batch

Copy code

Data store error:
No completed attempts of the task was found for task 'SportBasicDataGenerationFlow/1047/get_or_process_each_sample_keys/65282'

and

Copy code

[1047/get_or_process_each_sample_keys/65282 (pid 3335)] [pod t-wtfgs-n6h4l] Task finished with exit code None.

we have tried to increase memory and disk, but when we get the resources logs in graphana, that does not seem to be the issue... any suggestion?

ancient-application-36103

08/29/2023, 4:56 PM

Can you help with the full logs for this flow?

ancient-application-36103

08/29/2023, 4:56 PM

And what does your setup look like and any decorators on this step?

fast-pizza-24629

08/29/2023, 4:57 PM

we are using several of the decorators kubernetes, catch, environment. Let me see if I can find the logs

fast-pizza-24629

08/29/2023, 5:04 PM

would the 0.task_stderr.log
in the metaflow bucket for the task work?

fast-pizza-24629

08/29/2023, 5:12 PM

Copy code

...   
    @timeout(minutes=10)
    @catch(var="process_failure")
    @resources(gpu=1)
    @environment(vars={"GST_PLUGIN_PATH": "sunstone/gstreamer"})
    @kubernetes(
        disk=KUBERNETES_DISK,
        namespace=KUBERNETES_NAMESPACE,
        service_account=KUBERNETES_SERVICE_ACCOUNT,
        tolerations=KUBERNETES_TAINTS,
    )
    @step
    def get_or_process_each_sample_keys(self):
    ...

21 Views

Open in Slack

Previous Next