Hi! We're running a flow on GKE, using

nvidia/cuda:12.3.1-devel-ubuntu20.04

as the base image and using GPU's. When I'm running the flow remotely from my laptop I can't see any of the stdout output. Nothing I

print

in the flow or any other logging shows up. But if I find the pod it's running on and do

kubectl log <podname>

I can see the output. I have no idea how to troubleshoot this, I was wondering if anyone else have encountered this.

@salmon-smartphone-62113 - what are the contents of your step?

The plan is to do SFT on llama2, but I've paired things down to a very simple flow purely for debugging:

from metaflow import FlowSpec, step, kubernetes, resources, retry
import torch

class HelloCloudFlow(FlowSpec):
    @step
    def start(self):
        """
        The 'start' step is a regular step, so runs locally on the machine from
        which the flow is executed.

        """
        from metaflow import get_metadata

        print("HelloCloud is starting.")
        print("")
        print("Using metadata provider: %s" % get_metadata())
        print("")
        print("The start step is running locally. Next, the ")
        print("'hello' step will run remotely on Kubernetes. ")

        self.next(self.hello)

    @kubernetes(
        tolerations=[
            {
                'key': 'gpu_enabled',
                'operator': 'Equal',
                'value': 'true',
                'effect': 'NoSchedule'
            },
            {
                'key': 'nvidia.com/gpu',
                'operator': 'Equal',
                'value': 'present',
                'effect': 'NoSchedule'
            }
        ],
        node_selector={'gpu_enabled': 'true'}
    )
    @retry
    @resources(gpu=1)
    @step
    def hello(self):
        self.message = "Hi from the cloud!"
        print("Metaflow says: %s" % self.message)

        print("Running on Kubernetes with GPU.")
        gpu_available = torch.cuda.is_available()
        if gpu_available:
            print(f"Running on GPU.")
        else:
            print("No GPU available.")

        self.next(self.end)

    @step
    def end(self):
        """
        The 'end' step is a regular step, so runs locally on the machine from
        which the flow is executed.

        """
        print("HelloCloud is finished.")
        print("")

if __name__ == "__main__":
    HelloCloudFlow()

All the stuff I print in

start

and

end

steps are displayed, but I don't see any of the output from the

hello

step.

can you paste your console logs too?

I have the same issue. It's happening on this line: https://github.com/Netflix/metaflow/blob/cbf9b7f198bf2f1e255e0dda5c47324b63cc8bd3/metaflow/plugins/gcp/gs_tail.py#L49 download_as_bytes raising a 404 NotFound error after the object in the bucket is updated with the new logs. Changing the code to initialise the blob client object every time solves the issue. I'm not sure why this is happening though, wasn't happening before. Is it related to google-cloud-storage library or IAM permissions?

404 GET <https://storage.googleapis.com/download/storage/v1/b/testbucket/o/tf-full-stack-sysroot%2FSimpleTestFlow%2F18%2Frun_on_cpu_remote%2F274258%2F0.task_stdout.log?alt=media&generation=1709809091417399>: No such object: testbucket/tf-full-stack-sysroot/SimpleTestFlow/18/run_on_cpu_remote/274258/0.task_stdout.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)