acoustic-van-30942
01/25/2024, 2:21 AMKubeRay
and our first stab at it will be a tad hacky.
Here's an example:
@kubernetes(memory=2000, cpu=1)
@pypi(python="3.10.1", packages={"kubernetes" : ""})
@step
def setup_ray_cluster(self):
import tempfile
import json
from kubernetes import client, config
from kubernetes.client.rest import ApiException
print("Read in KubeRay Cluster Resource Config ...")
kuberay_config_file = tempfile.NamedTemporaryFile(mode="w")
kuberay_config_file.write(self.kuberay_cluster_config)
kuberay_config_file.seek(0)
with open(kuberay_config_file.name) as f:
kuberay_cluster_specs = json.load(f)
config.load_kube_config()
custom_api = client.CustomObjectsApi()
Running into an issue where the k8s pod cannot read the kubeconfig file (which I guess makes sense since the kubeconfig file is on a local machine), but I think we need to move it to the Metaflow K8s pod. Any insight on how to accomplish this?