https://outerbounds.com/ logo
Join Slack
Powered by
Hi Team, I was evaluating using metaflow as a pro...
# ask-metaflow
b
Hi Team, I was evaluating using metaflow as a production grade workflow orchestrator for ML workflows. I know metaflow interfaces with AWS step-functions. What are the backfilling capabilities which it provides. For example, i might have an hourly batch transformation job which consumes some data, performs a transformation and outputs a dataset. In case, the upstream dataset is not available and the step fails for certain hours. How easy is it to use the metaflow API and re-trigger those failed historical runs?
✅ 1
v
hi Muneeb! 👋 One pattern is to introduce a parameter, 
timerange
, which defaults to the current hour. You can schedule an hourly flow with 
@schedule(hourly=True)
and then it processes data hour by hour. Let's say it fails for five hours at night between 11pm-4am. You can trigger a backfill manually with
Copy code
python myflow.py step-functions trigger --timerange 20221213T23-20221214T04
your code would check if hourly data exists for the given time range and if it doesn't, it processes the missing data
b
Hi Ville, That's a nifty solution. Thanks for that. I had couple of more questions: 1. Suppose my flow has an hourly data processing step A and another step B which is daily. It depends on x hours of data outputted by A and x is a moving window. How do you suggest i handle this using using the metaflow steps. 2. Is it possible to create dependencies between two different metaflow flows. For example, an hourly flow of data-processing and then a daily flow which depends on some 'n' executions of that hourly flow ?
v
1. I'd create two separate flows, A and B. B uses the Client API to fetch A's output for the previous x runs. 2. Yep! See (1) 🙂
concretely, here's how you can fetch the latest 
x
runs of a flow 
A
Copy code
from itertools import islice
from metaflow import Flow
previous_x_runs = list(islice(Flow('A'), x))
you may want to add additional checks for the validity of results to account for missing hours, failed runs, backfills etc., but the basic idea remains the same
a cool thing about this approach is that you can make multiple parallel deployments of A and B using @project, e.g to test different versions of your flows, and the above snippet keeps different branches cleanly separated automatically
5 Views
Open in Slack
PreviousNext
Powered by