Hello folks, We are working on integrating with

KubeRay

and our first stab at it will be a tad hacky. Here's an example:

@kubernetes(memory=2000, cpu=1)
 @pypi(python="3.10.1", packages={"kubernetes" : ""})
 @step
 def setup_ray_cluster(self):
    import tempfile
    import json
    from kubernetes import client, config
    from kubernetes.client.rest import ApiException
        
    print("Read in KubeRay Cluster Resource Config ...")
    kuberay_config_file = tempfile.NamedTemporaryFile(mode="w")
    kuberay_config_file.write(self.kuberay_cluster_config)
    kuberay_config_file.seek(0)
    with open(kuberay_config_file.name) as f:
        kuberay_cluster_specs = json.load(f)
        
    config.load_kube_config()
    custom_api = client.CustomObjectsApi()

Running into an issue where the k8s pod cannot read the kubeconfig file (which I guess makes sense since the kubeconfig file is on a local machine), but I think we need to move it to the Metaflow K8s pod. Any insight on how to accomplish this?

could use

IncludeFile

if you want to pull it from the machine that runs the flow, or pull it from S3 in the

setup_ray_cluster

in case the user's machine might have a kubeconfig that changes or doesn't necessarily match the one you want KubeRay to read

OK thanks Eddie. Will give it a try