https://outerbounds.com/ logo
Join Slack
Powered by
Any idea why my flows get stuck when I try to run ...
# ask-metaflow
s
Any idea why my flows get stuck when I try to run them 
--with kubernetes
Copy code
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2024-05-16 15:42:15.873 Workflow starting (run-id 318):
2024-05-16 15:42:27.927 [318/start/1387 (pid 27208)] Task is starting.
2024-05-16 15:46:36.509 [318/start/1387 (pid 27208)] [job t-99448c87-ptnkk] Task is starting (Job status is unknown)...
I am using
Copy code
Metaflow 2.11.15
Tried to debug, but no luck with finding the underlying cause.
✅ 1
I dont think it is anything with the flow but minimal example,
Copy code
from metaflow import FlowSpec, step

class ExampleSubProcessTypesenseFlow(FlowSpec):

    @step
    def start(self):
        print("HELLO!")
        self.next(self.end)

    @step
    def end(self):
        print("BYE!")

if __name__ == "__main__":
    ExampleSubProcessTypesenseFlow()
My 
~/.metaflow/config.json
(masked)
Copy code
{
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_KUBERNETES_NAMESPACE": "default",
  "METAFLOW_SERVICE_INTERNAL_URL": "INTERNAL_URL",
  "METAFLOW_SERVICE_URL": "INTERNAL_URL",
  "METAFLOW_DATASTORE_SYSROOT_S3": "INTERNAL_S3_PATH",
  "METAFLOW_DATATOOLS_S3ROOT": "INTERNAL_S3_PATH",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "argo-workflow",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_KUBERNETES_TOLERATIONS": "[{\"key\": \"res-worker\",\"operator\": \"Equal\",\"value\": \"true\",\"effect\": \"NoSchedule\"}]",
  "METAFLOW_KUBERNETES_NODE_SELECTOR": "INTERNAL_K8S_CLUSTER_NAME",
  "METAFLOW_ARGO_EVENTS_EVENT_BUS": "default",
  "METAFLOW_ARGO_EVENTS_EVENT_SOURCE": "argo-events-webhook",
  "METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT": "operate-workflow-sa",
  "METAFLOW_ARGO_EVENTS_EVENT": "metaflow-event",
  "METAFLOW_ARGO_EVENTS_WEBHOOK_URL": "INTERNAL_ARGO_URL"
}
f
your cluster might not have enough resources? What kind of resource are you asking
s
As you see in the flow, I am not requesting anything special. I start the flow simply as 
python path/to/file.py run --with kubernetes
. i have one unused node in my cluster with 8GB RAM.
f
I think you need to add 
@resource
too as the decorator on these steps.
s
Are they mandatory? I hope not.
f
No.
do you have 
kubectl
locally installed? I would run 
kubectl get pods
and then 
kubectl logs <pod_id>
or `kubectl describe pod pod_id`see what's going on
s
Thanks for pointing me to check 
kubectl
. It was erroring out with 
error: exec plugin: invalid apiVersion "<http://client.authentication.k8s.io/v1alpha1|client.authentication.k8s.io/v1alpha1>"
. Fixed that and now the flow works as expected in EKS.
🙌 1
51 Views
Open in Slack
PreviousNext
Powered by