https://outerbounds.com/ logo
Join Slack
Powered by
<https://github.com/Netflix/metaflow/pull/892> in ...
# dev-metaflow
m
https://github.com/Netflix/metaflow/pull/892 in 
2.5.0
introduced changes to 
metaflow_extensions
Is there any kind of documentation for 
metaflow_extensions
and any roadmap for when it will become a public/stable feature?
✅ 1
f
I didn’t find any. But there is an example package which shows the directory structure for the package: https://pypi.org/project/metaflow-card/
besides packaging, as far as saw, api didn’t change.
💯 1
s
@dry-beach-38304 can help with more details.
d
I am out today but will reply this evening hopefully. For full extensions support it has changed quite a bit from before but should provide a superset of functionality. There is a repo with an example structure but it is not updated. I will update and ping when I do. Hopefully tonight.
Alex, what functionality were you using with the extensions so I make sure I cover that.
I should add that in terms of public/stable support, we (at Netflix) are the only users right now that we know of and before making it public we want to see how others are using it to see what parts are useful and what parts are not. So, do let us know what you use from it and it will help us make a more informed decision.
💯 1
I updated this: https://github.com/Netflix/metaflow-extensions-template. I am out this week though so I did this rather quickly to hopefully provide a bit more info. Do not treat it as gospel or anything. Feel free to come back and ask questions but hopefully it provides some useful information. Depending on what functionality you are interested in, I can provide more detail (limited this week but I’ll do my best). I will also add that https://github.com/Netflix/metaflow/pull/959 fixes some issues with the mechanism. I have to address a few comments on it before getting it merged but depending on what you need, you may need this patch as well. Again, I do want to stress that you are a bit in the wild-west of metaflow here so YMMV and this definitely does not provide the smooth well supported experience that metaflow typically provides (but hey, this is the dev channel). That being said, we use it internally at Netflix (well, I am currently moving to this new NS package — testing it now).
m
Thanks for the comprehensive answer Romain 🙏 If "I am out this week" means you're on leave then please don't reply to this until you're back 😅 --- The main drivers behind us exploring the "wild-west of Metaflow" are: • Support for packaging from parent directories (a large part of this use-case has been solved by metaflow now following symlinks). Our solution to this was to search backwards for a configurable project root file (e.g. 
setup.py
) in 
MetaflowEnvironment.add_to_package
and add files to the job package. • Being able to install system packages that aren't available on Conda without building/deploying/maintaining a Docker image - this is particularly useful for one-off experiments or during rapid iteration. Our solution to this was to add to 
MetaflowEnvironment.bootstrap_commands
such that scripts with a certain pattern (e.g. `preinstall.sh`/`preinstall-<flow_name>.sh`/`preinstall-<flow name>-<step name>.sh`) are checked for and executed. • Install libraries before a flow-file is run on a remote compute environment - this is useful to make mixins available without having to symlink or make mixins with dependencies available. • To have a 
@pip
decorator that respects conda environments during local development - if two steps running locally use the same conda environment but the first one makes installs with 
pip
then the second step will also run with those 
pip
installs (because they aren't captured in the hash of the conda environment). Our solution to this was to use 
task_pre_step
/`task_post_step` / 
task_post_exception
to trigger a steps conda environment to rebuild if it's running locally with 
@pip
. --- Having changed the structure of our 
metaflow_extensions
to match the new structure - it now works locally but on batch (with 
METAFLOW_DEBUG_EXT=1
) I get 
metaflow_extensions:    WARNING: ignoring all plugins due to error during import: No module named 'click'
I get the following error when using 
@conda_base
and 
@conda
on the same flow.
Copy code
RuntimeError: Conflicts in metaflow_extensions configuration files:
Packages metaflow-extensions[myorg], and /tmp/tmptsfjou66/metaflow_extensions/myorg define the same configuration module 'metaflow_extensions.myorg.plugins.mfextinit_myorg'
Packages metaflow-extensions[myorg], and /tmp/tmptsfjou66/metaflow_extensions/myorg define the same configuration module 'metaflow_extensions.myorg.toplevel.mfextinit_myorg'
The patch you mentioned - https://github.com/Netflix/metaflow/pull/959 - fixes this issue! 👍
d
Y'a. Sorry about that. It's getting better. I'll try to get 959 merged next week when I am back. Thanks for all the detailed use case. I'll probably have some comments/questions next week too.
Open in Slack
PreviousNext
Powered by