Me again. Trying out my shiny new Stack on AWS. Running tutorial 05:

% python 05-hello-cloud/hello-cloud.py run

I'm getting an error:

Kubernetes error:
    The @kubernetes decorator requires --datastore=s3 or --datastore=azure or --datastore=gs at the moment.

It's not clear how to pass the option. It doesn't want to be on the command line like:

python 05-hello-cloud/hello-cloud.py run --datastore=s3

Fails. And it's not an argument to the @kuberneties decorator:

@kubernetes(cpu=1, memory=500, datastore="s3")

Also fails. How do I incorporate this option? Thx.

A few options but it’s likely you are missing more than just the option to tell MF to use S3. • CLI:

hello-cloud.py --datastore=s3 run …

• You can set it up in a configuration file or through an env variable

METAFLOW_DEFAULT_DATASTORE

You likely also need the DATASTORE_SYSROOT_S3 setting set too. How did you setup your stack? You should basically have a file called

.metaflowconfig/config.json

that has all those settings. Metaflow will look in there by default.

Apparently I didn't setup my stack. I've been confused by "Setting up the Dev Stack", which I understood to be optional even when running on AWS. What I forgot about was "metaflow configure aws". I'd seen the configure step days ago and overlooked the sentence in "AWS Managed with CloudFormation" page. I just spent a while trying to run "metaflow configure aws". I think I figured out the bucket. But I cannot find the URL for either the Metadata service or AWS Batch. I think I need at least the lattter. Can you point me toward those URLs? Thanks.

ok, there may be a bit of confusion here: • Dev stack: local stack using docker and minio to emulate a AWS stack. You can definitely use that and everything will run locally. • An AWS stack (like the one for

metaflow configure aws

is more like this: https://docs.outerbounds.com/engineering/deployment/aws-managed/cloudformation/ which requiers an actual AWS cloud account to set up. Which one are you looking to do? (to run the demos, both will work, one is just running on your local machine and the other one is an actual env)

Hi Romain. Sorry for the slow response. I managed to get things going and I've been feverishly running fancier test cases. I'm making real progress. The relationship between the "dev stack" and what one gets from "metaflow configure aws" confused me. But in the end I've not installed the dev stack. I can still run locally and via Batch on AWS. I'm happy with that. I'm currently writing up some how-to documents for my colleagues. I may be in a position to send those out for comment. To whom would you suggest I send a markdown file or two? Thanks for your help!

You can past them on the #C020U025QJK channel. Sakari or Savin may be able to provide good feedback (or you can send to them directly). I don’t setup AWS stacks too much anymore 🙂

Thanks!! I'll do that.