Hi team, I have an

istio

related question: I’m invoking my flow locally via

poetry run python ./flow.py --environment=pypi run --with kubernetes

It creates the pod in my k8s namespace. But the pod is stuck in

NotReady

and metaflow never advances to next step. Pod describe shows that there’re 3 containers, the

start

step container completed but the

istio-proxy

container keeps running. It seems to me that metaflow doesn’t know how to shutdown the

istio-proxy

container and thus kept waiting. Any insights is appreciated. thankyou P.S. I researched a bit it seems by adding following to the pod may help.

annotations:
  proxy.istio.io/config: |
    terminationDrainDuration: 30s

But I’m not sure if this is the right way or how to inject this annotation when metaflow creates the pod.

yeah this can be a pure python decorator

Thanks for following up. Good to know about the annotation support. (But I couldn’t find the right annotation to use at this moment)

a pure python decorator

is there any good place for me to read more about how to achieve this? A naive way to put what I need is to inject a POST call: 1.

start

step metaflow container finished 2. TODO: inject a POST call here in the pod. 3. istio-proxy container shuts down in response to the POST call.

@step
    def start(self) -> None:
        # <http://requests.post|requests.post>('<http://localhost:15000/quitquitquit>', timeout=10) # NOTE: this won't work as it cuts off the network connection, and metaflow `start` container can't report success back to the metadata service
        self.next(self.end)
        # <http://requests.post|requests.post>('<http://localhost:15000/quitquitquit>', timeout=10) # NOTE: this isn't allowed by metaflow syntax