Hi there- trying to run a kubernetes job with a very simple

helloworld.py

, but it's getting hung up on

Task is starting (Pod is pending, Container is waiting - ContainerCreating)...

. The metaflow-ui is working on my cluster, and I can monitor jobs in the UI. Running the job locally also works. Trying to understand how to debug the k8s setup.

python helloworld.py logs 37/end

shows "No Tasks found at the given path -- either none exist or none have started yet", and I'm not entirely sure what the path is to test that my setup is working properly. I've also tried using the

@kubernetes

decorator, but the results are the same. Any advice?

❯ python helloworld.py run --with kubernetes
Metaflow 2.7.12 executing ParameterFlow for user:username
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2023-03-25 07:35:49.938 Workflow starting (run-id 37):
2023-03-25 07:35:50.418 [37/start/68 (pid 2055946)] Task is starting.
2023-03-25 07:35:51.522 [37/start/68 (pid 2055946)] [pod t-2fs2s-p2xjh] Task is starting (Pod is pending)...
2023-03-25 07:36:03.694 [37/start/68 (pid 2055946)] [pod t-2fs2s-p2xjh] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2023-03-25 07:37:52.445 [37/start/68 (pid 2055946)] Kubernetes error:
2023-03-25 07:37:52.445 [37/start/68 (pid 2055946)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-03-25 07:37:52.587 [37/start/68 (pid 2055946)] 
2023-03-25 07:37:54.148 [37/start/68 (pid 2055946)] Task failed.
2023-03-25 07:37:54.558 Workflow failed.
2023-03-25 07:37:54.558 Terminating 0 active tasks...
2023-03-25 07:37:54.558 Flushing logs...
    Step failure:
    Step start (task-id 68) failed.

helloworld.py

from metaflow import FlowSpec, step


class HelloFlow(FlowSpec):
    """
    A flow where Metaflow prints 'Hi'.
    Run this flow to validate that Metaflow is installed correctly.
    """

    @step
    def start(self):
        print("HelloFlow is starting.")
        self.next(self.hello)

    @step
    def hello(self):
        print("Metaflow says: Hi!")
        self.next(self.end)

    @step
    def end(self):
        print("HelloFlow is all done.")


if __name__ == "__main__":
    HelloFlow()