Our team has a cloud-based multi-tenant SaaS application that helps users determine where to best spend their maintenance budgets get the most out of making their facilities reliable. In our model, we have multiple tenant which contain facilities which contain units that are broken down into assets. Users configure dataset, including a few time series, some properties that are non-date/time effective. Within the scope of these individual assets we run a calculation pipeline which we currently have in Azure with service bus messaging and Azure Durable Functions for orchestration. This triggers a couple of downstream workflows to aggregate data up to the unit level and then up to the facility level. So all the data is specific to a particular ID value of some entity it is scoped to. We are looking at porting the calculation pipelines to Metaflow to gain the benefits of simpler versioning, visibility in Metaflow and Argo dashboards, debugging/resume support, and simpler management of our data science code and deployments. Other things that are important to us are individual calculation performance, scaling infrastructure up and down to control cost, error handling for long running workflow orchestrations, and traceability across many different composed workflows. One of my initial concerns has been startup time of steps, because we may have calculation pipelines kicked of for 1000 assets at a time each with 5 or so steps, so adding 10's of seconds of environment setup adds up for our use case. I know @User has been doing some excellent work in this area and we can alternatively manage our own docker images per metaflow with python and dependencies pre-configured to mitigate this, so this is less of a concern now. My real question is what are we not yet seeing that could be a blocker for our use case? We've noticed all examples we've seen are around having a single large data set and needing to process through this dataset periodically in bulk rather than the multi-tenant dataset per entity type of use case. Are we looking at the right framework? Would Metaflow be a good fit for what we are trying to achieve? I am not looking to make Metaflow fit my specific cases perfectly as I know it needs to be general purpose, but mostly need to know if we are both heading in 2 fundamentally different directions. Thanks for any insights you are able to provide!

Hey Ben! 👋 Have you checked out Metaflow docs on the foreach pattern? If the dataset is all grouped in one bucket you can imagine using the ID of one entity to chunk the data. Alternatively, if the data is not stored all together, you could split your foreach using a list of URIs to each dataset. Then you can do downstream stuff like hyperparam tuning uniquely foreach of the these parallel processes. This can also work hierarchically, if you want to split at several levels and then aggregate results by level in post-processing steps. Maybe something like the following DAG structure could address your case?

start --> 
  foreach facility --> 
    foreach unit --> 
      foreach asset --> 
        tune model or run calculation pipeline for (asset, unit, facility) -->       
      aggregate asset results within each unit --> 
    aggregate unit results within each facility --> 
  aggregate facility results --> 
end

Thank you. This is quite an interesting way of thinking of it @crooked-jordan-29960. We have always been in a mode of thinking a calculation pipeline needs to be individually triggerable by an event as all asset data sets have a different rate of change and a user may make changes to an individual asset, click calculate, and then the results from that one asset need to come back to the user in minutes. But maybe if we are smart about keeping track of which facilities, units, assets have changes to their data sets, there could be one overall pipeline does the nested foreach and only executes for facilities, units, and assets that need to be updated. If a user clicks calculate it checks across the hundreds of thousands of assets and maybe only that one asset changed, so it could be quick to go through the pipeline run, as long as we are properly debouncing this user event that triggers the calc when there's higher user load. If each asset is producing around 4GB of time series and other output data would there be a memory concern if we are just attaching each of these to a nested foreach array type thing on the

self

of the top level flow. Does metaflow know to maybe keep the other datasets on disk or something until I try to read them with self.facility[22].unit[33].asset[638]? The other direction I was thinking was breaking down into smaller calculation pipelines where an individual run for an asset may have a tag appended that has the type of calculation and an entity id, like

asset-risk-638

or

asset-risk-875

and then when a downstream

unit-aggregate-33

runs it knows that it needs to read results from child flows,

asset-risk-638

or

asset-risk-875

using the client sdk to access last run of these flows in the proper namespace. But it seems like this would add a bit of additional complexity.

Makes sense. My earlier suggestion about the foreach pattern seems a bit of a roundabout way to get to the end goal in this case where any given asset might trigger a flow run independent of the other assets. It may be easier to maintain and analyze flow results with some combination of the more focused/asset-level pipeline idea you mentioned at the end, and/or parameterizing a flow that does the calculation pipeline for one asset, but takes in general runtime parameters that can be used across many of the assets. Using tags could be nice in this approach. At some threshold of the number of ways the same flow can be parameterized it becomes required to have some metadata s that helps you filter runs when analyzing results. • Does metaflow know to maybe keep the other datasets on disk or something until I try to read them? (note response to this question is mainly applicable to the foreach approach) ◦ It doesn't exactly store things on disk automatically where you could access in a chained call like self.facility[22].unit[33].asset[638], but you can still access the data downstream in a similar way. A caveat is that it might be more efficient at 4GB of each dataset to use tools like Metaflow's S3 client to read and write as opposed to sharing state across steps using

<http://self.my|self.my>_data

variables. Anyways, when using the foreach structure, Metaflow requires that you define some logic on how to merge the

<http://self.my|self.my>_data

variables from upstream steps. These are often referred to as join steps in the documentation and are required after foreaches. There is a

merge_artifacts

utility that can help you pick and choose exactly which slices or sets of 4GB data would need to read into memory or disk for the downstream task. Memory requirements will definitely blow up in the aggregation steps if you try to merge hundreds of 4GB datasets like this in one step, which is totally doable with Metaflow, just more of a cost/speed consideration at that point.

Thanks, @crooked-jordan-29960. Shortly after this conversation I had discovered the https://outerbounds.com/docs/whether-to-self/ article that helped in my understanding on this also. I guess if I'm in Azure the S3 library would need to be configured to work with minio. I've also enumerated a few other options I'd like to look into to build out our non-existent data layer that would be used for input and output of the flow. I've attempted to add a guess at order of magnitude of eventual cost of a small to midsize company's usage. 1. Snowflake (Post-Modern Data Stack) ($ to $$$ depending on how fresh you want your data) 2. Parquet files on Azure Data Lake Storage accessed with either DuckDB or Apache Arrow (support for chunking) ($) 3. Azure Synapse with Azure Data Lake Storage ($$) 4. Metaflow's S3 library with Minio storing to Azure SSD, Blob, and Cool Blob ($) 5. Databricks Data lakehouse or the Azure variant ($$) 6. Data in Tables (Hive, Spark, Postgres, Azure SQL, etc) ($ to $$) 7. Cognite Data Fusion as the data layer (???) 8. Many others...

This is awesome, Ben! Thanks for sharing your research, have seen lots of activity lately around aspects of the Metaflow data layer topics you mentioned. Also FYI, Metaflow now has native support for Azure, so Azure blob storage is an option too (should be mostly parallel to the S3 client from UX point-of-view) if you aren't already committed to minio. Feel free to DM if you'd like to set some time to talk through the decision space on a call some time 🙂