I have a metaflow workflow, I want to call the workflow repeatedly whenever I read 6000 rows from a database. How can I do this? Also I want to pass the 6000 rows dataframe to the workflow and the entire workflow and infrastructure is on my local machine

Why call it repeatedly and not run some batched loop inside the flow itself?

Actually, in the production the whole metaflow code needs to run from start to end for every 1 minute of data file having granularity of 10ms. Can you please share some reference of this batched loop?

You have fixed size data, right? Not streaming or something?

I have timestamp based data, and the data will be coming in real time. I have the columns size fixed but the rows I have to keep storing it in a buffer until I get data for complete 1 minute. As soon as I get the data, metaflow workflow should be called for that particular 1 minute data, and this process will keep on going. As of now I have used sub-process to do this, is there any better way to do so?

Do you mean you call the trigger command from a subprocess? That's ok. I don't think there is a trigger API. The other thing you can do is to use some event based mechanism if you use Argo workflow. For every one minute you can create an event with start and end timestamp as params which then triggers your flow.

No I am not using Airflow, that's the main issue

I didn't say you need airflow. I asked about the argo workflow. What are you using then?