Hi, I've noticed that it is not possible to run flows locally when you have the

kubernetes

decorator specified on a flow. (It complains about the need for a remote datastore.) I'm a bit surprised by this, so I was wondering if someone could explain the reasoning behind this error? I thought one of the points of Metaflow was to be able to run the same code locally and remotely, so was surprised that this decorator stops one from running a flow locally. Naively I'd expect this check to happen only when the step is actually run on Kubernetes, not when you run the flow locally.

Hey Thomas... thanks for the question! When a step has the

@kubernetes

decorator, it is run as a pod on a K8s cluster. Metaflow needs to package the code, data, config and parameters and make them available in the pod so that they step runs correctly. Metaflow uses a remote datastore for storing these. So, basically the sequence of events is: • metaflow sees the

@kubernetes

decorator • packages code, data, config and paramters and writes to the a task specific location in the remote datastore • starts a pod in K8s • inside the pod, the packaged code, data, config and params are downloaded from the remote datastore and setup appropriately • code from with

@step

is executed

OK, I see. Perhaps I should rephrase my question a bit, why is this behaviour determined by the

@kubernetes

decorator and not the

--with kubernetes

flag? It just seems to go against one of the main ethos of Metaflow to preclude people running code locally, so I'm still a bit confused why it is configured in this way.

I believe the functionality is similar to @batch and —with batch, where the decorator allows you to target specific steps to run on batch and the —with cli command applies to the entire flow

+1.

--with kubernetes

on the cli will run all the steps in the flow on Kubernetes. With

@kubernetes

decorator, you can target a specific step to run on K8s.

I see. Makes sense. What is the intended workflow for people that want to be able to run a flow both locally and remotely on Kubernetes? I'm currently using the

resources

decorator, which works fine for this use case. However, I have the feeling that there is some functionality missing from this decorator that is not missing from the

kubernetes

decorator, e.g., disk space. (I might be wrong on that, or maybe it is a typo in the docs, but from the docs the APIs look different.)

Typically, the sequence I've used is like this: 1. run the flow locally and ensure that it works as expected:

python flow.py run

2. run the flow on K8s:

python flow.py run --with kubernetes

and verify that it works as expected. This might be slightly inefficient since all steps, including the steps that don't need to, will run on K8s (e.g.

start

and

end

steps). You could also specific additional config options

--with kubernetes:image=my_docker_image,cpu=8

3. once it is confirmed that the pods in K8s work correctly, you can add the

@kubernetes

decorator to specific steps so that only they run on K8s.

python flow.py run

(will need to also run

metaflow configure

first to configure a datastore or run with

METAFLOW_DATASTORE_SYSROOT_S3

env variables). 4. Another option here is the simply make the flow into an Argo workflow and trigger it as and when needed:

python flow.py argo-workflows create

and

python flow.py argo-workflows trigger

. The entire flow will run on K8s as an Argo workflow. 5. If the goal is to run the flow periodically, use the

@schedule

decorator and then create the Argo workflow. It will get triggered as per the

@schedule

.

This is similar to a flow I am using. I actually go straight to Argo Workflows directly for remote runs. I also don't use the

@kubernetes

decorator, but the

@resources

decorator instead. One of the issues I have with the

@kubernetes

decorator is that if I want to go from step (5) to step (1), e.g., I want to do a new iteration on a production model, then I would need to comment out the

@kubernetes

decorator to iterate on the flow locally. Similarly, if I want to do unit tests of the flow in my CI then I would likely not have access to a remote data store, so it is not clear how I should proceed if I use the

@kubernetes

decorator. I'm happy to use the

@resources

decorator. My question was more, is this the intended use case for this decorator?

Makes sense. You're right. Using

@kubernetes

may work on specific envs. Use

@resources

if it's working fine for you. There are some options that

@kubernetes

supports that

@resources

does not (e.g.

image

,

namespace

, etc.). See resources vs kubernetes. If you want to run flows as part of a CI and have

@kubernetes

.. a couple of options would be: • run the CI job with specific variables such as

METAFLOW_DATASTORE_SYSROOT_S3

• run the CI job with metaflow config file (which has the config options and values). This might help.