Hello everyone, I’m new to metaflow and I’ve been experimenting a few things with it. As I understand it, you would be able to run some steps locally and others on the cloud even though they belong to the same flow. Let’s say that I use the start step to generate artefacts for my ml task and I decide to run it locally. The following step does some heavy lifting and I want to run it on the cloud. How am I able to retrieve the artefacts I have created previously since my Kubernetes cluster do not have access to my local metaflow datastore ? What are the good practices in this kind of setup ? Thanks !

Even though you are running that step locally, data artifacts get persisted in a data store such as AWS S3. The downstream step in the cloud when referencing the data artifact will load the data in from the S3 bucket. The S3 bucket is the one set within

METAFLOW_DATASTORE_SYSROOT_S3

and

METAFLOW_DATATOOLS_SYSROOT_S3

, assuming the

METAFLOW_DEFAULT_DATASTORE

is set to

S3

.

+1 to @ambitious-bird-15073. @blue-vegetable-65043 you would need a blob store that's accessible by both your laptop and kubernetes - this can be S3, GCP, ABS, Minio

Yes, you’re right that makes sense. I thought I had a datastore problem but that’s not the case. Let me give you more context. I have a start step that looks like this:

def start(self):
    ...
    self.folds = []
    for train_index, test_index in tscv.split(self.df_tweaked):
        self.folds.append((self.df_tweaked[train_index],self.df_tweaked[test_index],))
    self.next(self.foreach_model, foreach="models")

And the step foreach_model looks like this:

@pip(libraries={"scikit-learn": "1.2.2"})
@kubernetes(memory=12000, cpu=4)
@step
def foreach_model(self):
    self.model = self.input
    self.next(self.foreach_fold, foreach="folds")

And when I run it I get the following error: Invalid self.next() transition detected on line 69: 35/foreach_model/312 (pid 62639)] [pod ] Foreach variable self.folds in step foreach_model does not exist. Check your variable. What Am I missing ? When I was running this 100% locally I had no issue. I’m puzzled.

can you help me with the contents of L69 -

Invalid self.next() transition detected on line 69

@square-wire-39606 I solved my issue, it wasn't related to metaflow. Thanks for the help !