Hi there- trying to run a kubernetes job with a very simple

helloworld.py

, but it's getting hung up on

Task is starting (Pod is pending, Container is waiting - ContainerCreating)...

. The metaflow-ui is working on my cluster, and I can monitor jobs in the UI. Running the job locally also works. Trying to understand how to debug the k8s setup.

python helloworld.py logs 37/end

shows "No Tasks found at the given path -- either none exist or none have started yet", and I'm not entirely sure what the path is to test that my setup is working properly. I've also tried using the

@kubernetes

decorator, but the results are the same. Any advice?

❯ python helloworld.py run --with kubernetes
Metaflow 2.7.12 executing ParameterFlow for user:username
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2023-03-25 07:35:49.938 Workflow starting (run-id 37):
2023-03-25 07:35:50.418 [37/start/68 (pid 2055946)] Task is starting.
2023-03-25 07:35:51.522 [37/start/68 (pid 2055946)] [pod t-2fs2s-p2xjh] Task is starting (Pod is pending)...
2023-03-25 07:36:03.694 [37/start/68 (pid 2055946)] [pod t-2fs2s-p2xjh] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2023-03-25 07:37:52.445 [37/start/68 (pid 2055946)] Kubernetes error:
2023-03-25 07:37:52.445 [37/start/68 (pid 2055946)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-03-25 07:37:52.587 [37/start/68 (pid 2055946)] 
2023-03-25 07:37:54.148 [37/start/68 (pid 2055946)] Task failed.
2023-03-25 07:37:54.558 Workflow failed.
2023-03-25 07:37:54.558 Terminating 0 active tasks...
2023-03-25 07:37:54.558 Flushing logs...
    Step failure:
    Step start (task-id 68) failed.

helloworld.py

from metaflow import FlowSpec, step


class HelloFlow(FlowSpec):
    """
    A flow where Metaflow prints 'Hi'.
    Run this flow to validate that Metaflow is installed correctly.
    """

    @step
    def start(self):
        print("HelloFlow is starting.")
        self.next(self.hello)

    @step
    def hello(self):
        print("Metaflow says: Hi!")
        self.next(self.end)

    @step
    def end(self):
        print("HelloFlow is all done.")


if __name__ == "__main__":
    HelloFlow()

Can you check 37/start?

Same thing

python hello.py logs 37/start
Metaflow 2.7.12 executing HelloFlow for user:username
    Invalid command:
    No Tasks found at the given path -- either none exist or none have started yet

Metaflow 2.7.12 executing HelloFlow for user:username
    Invalid command:
    No Tasks found at the given path -- either none exist or none have started yet

Easiest way to debug would be to inspect the pod directly (

t-2fs2s-p2xjh

)

e.g.

kubectl get pods --all-namespaces | grep t-2fs2s-p2xjh

This will let you know the k8s namespace

then

kubectl describe pod -n NAMESPACE

,

kubectl logs t-2fs2s-p2xjh -n NAMESPACE

, etc

Ah, ok - so it looks like it it doesn't like my S3 credentials?

❯ kubectl logs t-2fs2s-p2xjh -n metaflow
Setting up task environment.
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Failed to download code package from <s3://metaflow-sandbox/metaflow/HelloFlow/data/30/3053e0326438eb7ecdde591f9fd6fa6efb822884> after 6 tries. Exiting...

AWS_SECRET_ACCESS_KEY

in the config json is never propagated to Kubernetes pods. When you first start the job (from local terminal), MF will write some initial bookkeeping info (pre kubernetes job submission). That is what you are seeing in s3. This means that your local env has sufficient perms to write to Wasabi (but it did not come from the config json).

We also need a way for the Kubernetes pods to have perms to access Wasabi.

AWS_ACCESS_KEY_ID

is not propagated from config.json.

You can inject adhoc env vars using the

@environment

decorator

Alright. I'll take a look at this. Thanks!

Recommend looking at https://docs.metaflow.org/api/step-decorators/kubernetes (secrets). You may not even need environment decorator, with that.