Hi Everyone!!
I'm currently working on a migration project and therefore created a utility that converts the list of python functions into metaflow steps that fills up placeholders in a Metaflow template. After running the utility its creating a very big flow containing approximately 950 steps. Some steps will be executing in parallel while others will run in a sequential manner. I've redacted the step names and trimmed it to contain around 300 nodes. Also each step just points to the next step. Please try to execute this flow in your machine.
In my project, after running multiple experiments, I've found out that , Metaflow starts lagging as soon as the number of steps increases beyond 125 steps and refuses to start beyond 300 steps.
I'd appreciate if you could provide a way to execute these flows.
I tried executing the flow in a decent machine with 16 cores/64GB RAM, and during the startup its not utilizing more than 20% of its resources