Hello all, found out about Metaflow recently. I have a use-case where messages come in via some kafka topic. These messages need to be transformed and dropped into another topic. I really like the simple framework in Metaflow to define workflows. I also like that the steps are recording everything and displays in a nice UI. I wonder if Metaflow is an overkill for this purpose.

@some-beard-80086 Metaflow is a better fit as a batch processing pipeline - if you want to transform messages stored in your broker en masse, for example.

is it because of worry of latency could be longer?