Guys! Hey! I try my best to set up data storage with on-prem s3 and local Metaflow service. I used this code:

from metaflow import FlowSpec, Parameter, step

import os
os.environ['METAFLOW_DEFAULT_DATASTORE'] = 's3' # or it should be 'local'?
# os.environ['METAFLOW_DATASTORE_SYSROOT_S3'] = 's3'


class ParameterFlow(FlowSpec):
    alpha = Parameter('alpha',
                      help='Learning rate',
                      default=0.01)

    @step
    def start(self):
        print('alpha is %f' % self.alpha)
        self.next(self.end)

    @step
    def end(self):
        print('alpha is still %f' % self.alpha)

if __name__ == '__main__':
    ParameterFlow()

My CLI command from here (https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-airflow#pushing-a-flow-to-production):

python parameter_flow.py --with retry airflow create parameter_dag.py

I got this error:

Airflow Exception:
    Datastore type `local` is not supported with `airflow create`. Please choose from datastore of type `azure`or `s3`or `gs` when calling `airflow create`

I still don't quite understand what I need to change. Sorry guys! And thank you in advance!

Did you try running

metaflow configure

?

Oh, nope. Let me figure it out. I didn't even know about this lol.

That will ask you bunch of configuration options. Yeah it's bit hidden from tutorial

For instance, if I want to work with kuberenetes, what preliminaries should I consider before launching this simple toy example?

I am not really well aware of the local set-up either. I use EKS with outerbound provided terraform config and it writes the config file. So you will just have to try and see how it goes.

Thank you so much! I'll try my best haha

the config.json from Metaflow does not replace environment variables per se. As such, setting the

AWS_DEFAULT_REGION

should not have any effect through the Metaflow config, as that is a value used by the botocore library. Either setting this as an environment variable before running the command, or configuring it as part of your AWS profile in

~/.aws/config

should do the trick.

Thank you! I'll try.

One additional note - most Metaflow variables (IIRC) are prefixed with METAFLOW_*. Using the metaflow configure CLI will be helpful therefore than directly overriding the json.