Hello! Brand new Metaflow user here with potentially boneheaded question. I'm hoping to use Metaflow on a long-running petabyte-scale data transformation job with potentially up to a couple million steps/tasks depending on how I chunk my manifest. (I've used similar approaches with Step Functions DistributedMap & ECS Fargate before, so feel like Metaflow will be able to handle this, but please correct me). The DAG is a simple fanout to workers, with no additional joins / reductions necessary. My question is, since I don't have any need to interrogate step artifacts after foreach/map branches complete, how should I handle my

join

? Is it absolutely required, or is there some type of no-op I can specify to avoid loading a million artifacts from the datastore if I don't need to? Am I missing something / thinking about this incorrectly?

the artifacts are always lazily fetched, so if your join step doesn't reference them, you should be all good.

Welcome to Metaflow!

couple million for-each tasks should work out well - we tested at that scale when we integrated with distributed map

Perfect, thank you! While I have your ear, I've also stood up the metadata service & UI infrastructure. With this many tasks, should I be thinking to scale up those services and the postgres backend DB as well? If so, any heuristics you can share on right-sizing those?

technically you don't strictly need the metadata service, but it is useful for tracking etc. you can use aurora severless v2 so that you don't have to worry about scaling the db - the metadata service has a tiny footprint and doesn't do much work by itself, having a couple of replicas should be enough

🫡 Appreciate it! Excited to get this going and hopefully introduce Metaflow to my data engineering/data science team! We're hand-rolling all of our big batch processing jobs now and Metaflow seems like a really great way to simplify our operations. Thanks for all of your work here everyone!

exciting! yeah, let us know if you run into any issues