Hello team,
Some of our users have expressed inte...
# ask-metaflowa
acoustic-van-30942
11/01/2023, 6:07 PMHello team,
Some of our users have expressed interest in leveraging Metaflow for feature engineering pipelines? Would you happen to have examples/documentation that delves into more complex feature engineering using Metaflow?
This is a good start but looking for more resources - https://outerbounds.com/blog/developing-scalable-feature-engineering-dags/
c
crooked-jordan-29960
11/02/2023, 8:32 PM@flaky-plumber-70709 could use your feature eng+store wisdom! Any good examples you can point to / share?
crooked-jordan-29960
11/02/2023, 8:35 PM@worried-mechanic-36312’s famous repo has a dbt + great expectations pipeline that is pretty advanced - may be a good place to start
a
acoustic-van-30942
11/02/2023, 8:36 PMYeah I saw that too. Is that the recsys flow? That was a good start too but I think we are looking for a more complex feature engineering pipeline.
c
crooked-jordan-29960
11/02/2023, 8:36 PMwhat kinds of challenges is the team facing around feature eng?
a
acoustic-van-30942
11/02/2023, 8:38 PMGonna copy and paste what a member from my team is looking for:
> Would you be able help me ask Outerbounds team that if there is any example (code) or use case, using Metaflow directly for doing feature engineering?
>
> I am looking for some framework that user can easily define transformation steps as functions and chain them together as DAG, and showing lineage. That being said, it’s like micro-orchestration framework which can be scheduled and triggered by macro-orchestration tool like Airflow. I feel the functional requirement I mentioned here can be achieved by metaflow, how do you think?
acoustic-van-30942
11/02/2023, 8:41 PMFor feature engineering (ml use cases), sometimes the user may fetch different data resources (s3, kafka or snowflake), and do the transformation individually and merge together to transform a new feature set.
Of course, they can create a one gigantic processing code to handle all the logics from source to sink (e.g. offline feature store).
So usually we will encourage users to divide transformations logic into multiple smaller steps and construct as a DAG and using orchestration tool like Airflow.
Okay, then there is another issue, for ML use case, they may change their feature transformation logic very quickly, like branching out the same column and applying another transformation or creating another feature based on the same column, and also it’s really hard for users to figure out what the DAGs looks like in the beginning, so that they cannot even construct theirfiledag.py
acoustic-van-30942
11/02/2023, 8:42 PMSo basically in a nutshell, Airflow is not ideal because it's harder to do local runs for testing out the feature engineering pipeline. But Metaflow prioritizes local run in a first class manner, so will make it much easier for users to compose their feature engineering pipelines, so want to investigate Metaflow, but need strong examples to make this a compelling use case.
c
crooked-jordan-29960
11/02/2023, 8:45 PMso unlike the dbt or hamilton cases where it is more like the micro-orchestration inside a single metaflow step, you want the feature eng dag to be a subgraph in the metaflow dag - is that accurate?
a
acoustic-van-30942
11/02/2023, 8:58 PMWe are open to any pattern. Basically just want to peruse some well-defined dedicated feature engineering DAGs using Metaflow. 🙂
f
flaky-plumber-70709
11/02/2023, 8:58 PMso not really an answer but a lot of these use cases would be a lot easier if metaflow had integration w openlineage
flaky-plumber-70709
11/02/2023, 9:01 PMhaven't looked at it in a second but I think that would allow data to be tracked across diff producers (incl metaflow) and then scooped up in something like datahub / admundsen
flaky-plumber-70709
11/02/2023, 9:03 PMI got my team to punt on feature stores for now since there's a lot of legwork to be done w dbt, when we revisit I think dbt + materialize will do most of the heavy lifting and then maybe put a feature store ontop for the last mile features around governance
c
crooked-jordan-29960
11/02/2023, 9:08 PMnice context, thanks BG!
🎉 1
a
acoustic-van-30942
11/02/2023, 9:16 PMThanks Bryan!
a
ambitious-bird-15073
11/04/2023, 1:57 PM@acoustic-van-30942 have you checked out Chapter 9 in @victorious-lawyer-58417’s book? It covers feature engineering, here is the example code.
a
acoustic-van-30942
11/05/2023, 6:07 AMWill take a look, thanks @ambitious-bird-15073