Hi. I've configured things to use the AWS Batch facility. Since then, I've tried to go back and rerun tutorial 02-statistics. It now fails complaining about initializing S3:

Including file .../metaflow-tutorials/02-statistics/movies.csv of size 191KB 
    Parameter field failed:
    When evaluating the field default for the Parameter movie_data, the following exception occurred:

...
    metaflow.plugins.datatools.s3.s3.MetaflowS3URLException: Initialize S3 with an 's3root' or 'run' if you don't want to specify full s3:// urls.

I made a stripped-down version of the tutorial python script (attached). When I run the local command:

python 02-statistics/stats_short_includefile.py run

The failure above occurs. If I comment out the "IncludeFile" call. The job runs successfully. The overarching question is, now that I have AWS Batch set up, how do I cause a script to use a local data file? (I've also attached "movies.csv" to make it easier to run the test.)

In general it's not advised to bundle large files such as csv's into the zip. But if you really want to, there's `package-suffixes`: https://docs.metaflow.org/scaling/dependencies/project-structure#non-python-dependencies

Thank you, Nissan!! Your pointer to S3ROOT was on the money. I looked at my ~/.metaflowconfig/config.json file and the definition of S3ROOT had a leading "M". As in:

"METAFLOW_DATATOOLS_S3ROOT": "<Ms3://bpw>...

It must have been a typo by me when running

metaflow configure aws

. Now, both local and

--with batch

work just fine. And, thank you for the link to "Non-Python dependencies". The

--package-suffixes

option looks very useful and convenient. I'll be mindful of your "not advised to bundle large files" comment. You asked whether I'm running with

--with batch

. I am. Do you have any general comment or guidance on that approach? It seems much easier to set up. Thanks again!!

Were you already on AWS? Also how long are your jobs taking, and what sort of resource requirements do they need? if you're just looking to run in the cloud with minimal setup, I would actually suggest taking a look at https://github.com/npow/metaflow-sandbox (one of the things I've been working on to make it easier for folks to get started using metaflow in the cloud)

Hi Nissan. You ask a good question. My task has been to find and learn a toolset that would allow us to move large computations (e.g. simulations, ML, analysis of large datasets) to "a" cloud. (we're a company of about 100 with a large ratio of scientists, so lots of whacky computing tasks.) I came to Metaflow via an attempt to use terraform directly. The latter was not pleasant. I actually engaged Claude in a very satisfying discussion and it suggested Metaflow. For the most part, Metaflow has been very successful in the sense that I've been able to get (admittedly toy) problems to run locally and then moved to AWS. I've had a few issues, but this community has been very helpful and tolerant of a rather rank amateur. The one general issue I've had is that the documentation seems to suffer from what I'll call "mutipath", that is, there are a number of documents that are similar yet give slightly divergent information. This has caused me to miss key steps in bringing things up. I will say that I got a lot of satisfaction and encouragement just from running locally: it gave me the feeling that a) the methods for constructing codes (e.g. Flows, Steps, Joins, etc.) were very easy to learn and use. That motivated me to muscle my way thru getting the cloud parts to work, which I think I've been successful at (with help!). Our compute needs are varied. But one advantage of Metaflow is that one can develop locally, using the tools for even small problems. This A) gives one practice at using the tools; and B) is switchable to the cloud if the task gets bigger. The first point: keeping in practice, is actually very important. If the tool was only usable for work in the cloud, then any one of us would get only intermittent practice and one's skills would likely atrophy. Instead it'll be "All Metaflow all the time". Your pointer to sandbox is valuable. I had not thought to use it, and in thinking about why, two things came to mind. First, for whatever reason, I interpreted the term "sandbox" to mean "useable as a learning tool but not for real work". Second, my experience with Colab taught me that one must be careful about privacy: we work with a lot of client-proprietary data and almost always online or cloud tools are suspect if only by our clients. (Or clients being suspicious is enough for us to have to say "no".) I gather from you making the suggestion that the first point above is not correct: the tool is sufficiently capable for doing real work at some scale. The privacy question is just a matter of doing some research on your policies. I'll definitely look into both your policies and how to interface to the sandbox. One further thought, while I have your attention is that, as a "casual" user, I'm interested in condensed or "boiled-down" directions for getting things to work. (The "multipath" comment above is relevant.) I've already started a document to show our internal folks how to get started. Assuming I can get permission from the company (again, proprietary data questions) I'd like to offer what I come up with back to the project. (An existing example of this is that I chose CloudFormation to set things up. People here said that the CF method is less used and less tested, and that if I come up with any fixes or clarifications to put in a PR. I'm still hoping to respond helpfully to that request.) If you have a suggestion about how to supply information to the project. Please let me know what you think. I've gone on too long. But Thank You very much for your (and others') help. This environment is much better that Stack Overflow! Again, lemme know what you think.

I actually engaged Claude in a very satisfying discussion and it suggested Metaflow

Claude does seem to like us! We were on the Anthropic homepage at one point 🙂

there are a number of documents that are similar yet give slightly divergent information.

this isn't good. if you can point out which ones, we can try to make it clearer

one advantage of Metaflow is that one can develop locally, using the tools for even small problems

local -> cloud scaling is one of the core value props of metaflow: https://docs.metaflow.org/scaling/introduction

interpreted the term "sandbox" to mean "useable as a learning tool but not for real work"

I could see why you have that interpretation, although the pure definition only means isolation and restricted permissions, not a specific environment like dev/test/prod. To be clear, Metaflow does not have any opinion on your choice of cloud or backend provider -- it's up to you to pick the right ones to satisfy your requirements. That being said, if you have long running jobs then sandboxes such as daytona/e2b are likely not the right match for your workload since their core value prop is in fast container startup times (along with the isolation) -- if your job takes a long time then the startup time isn't that much of a problem. The only reason I pointed you to that project is because the setup is much less than wrangling with CF/Terraform to get things setup on AWS -- things you will probably eventually need anyway as your workload scales. I will also mention (not affiliated in any way) that Outerbounds has a managed cloud offering that's supposed to make the onboarding easier, although since you mention privacy concerns I don't know if that would be an acceptable solution for your business.

I'd like to offer what I come up with back to the project

We welcome contributions! Please start a thread in #C020U025QJK and we can take things from there