there are many valid use cases for wanting to find...
# dev-metaflow
s
there are many valid use cases for wanting to find an existing run based on its artifact values. @jolly-teacher-26015 brought up one such use case recently https://outerbounds-community.slack.com/archives/C02116BBNTU/p1622722846159800?thread_ts=1622637945.140300&cid=C02116BBNTU
but given that this is not obvious at the first sight and the need is pretty common, it might make sense to have a top-level convenience function like
find_run(flow, {'artifact': 'value'})
that essentially wraps the above list comprehension
🔥 2
I am curious to hear if anyone here has other use cases for
find_run
one concern is the performance of such function: The list comprehension is sequential so it fetches artifacts from S3 sequentially, which can be slow. It would be much more efficient to use
S3.get_many
to parallelize the operation but it makes the implementation a bit more involved. On the other hand, parallelization/performance is a great reason for providing such convenience function since no one wants to do that by hand.
👍 5