Hi! Is there a pattern/best practice for using a shared library across multiple flows? We've developed a bunch of utility functions, and are constructing a single repo for all the data science work, The idea is that there will be a directory for these shared utility functions, then a directory containing directories for flows attached to projects. Something like:

main dir
|-dist
|-utility dir
+-projects dir
  |-project 1
  | +-flow directory
  | project 2
    +- flow directory

We want the flows to be able to use the library in the utility directory, and are looking at having the utilities being built as part of makefiles in each project directory. The root level makefile will create a python package and place it into the

dist

directory to be locally available. What is the appropriate/correct/accepted method for having those utility functions get into the code that's sent to remote compute? Will the imports inside step functions "just work", or will we need to be using some internal pypi/conda distribution mechanism?

There are a few solutions here: • you can point back to

utility_dir

from all your flow directories. The packaging will work appropriately. This won’t work “out of the box” though if there is an actual “build” step (ie: it’s not just python files). You can probably also make it work by including more file type (with the --package-suffixes option) but may not be the most convenient. • you can package your utilities as a metaflow extension. It’s a bit of a simplified use of that mechanism but the idea is that if you publish your utilities package as a metaflow extension, if users have that installed when they run metaflow, it will get packaged up and sent over when executing remotely. This is not an officially supported path but I don’t see it changing (we use that extensively). If you have interest in that path, let me know and I can provide more details. Generally though, this is not something we do very well. We’ve toyed with a few ideas but never really did anything yet. It’s something that is on our (distant) radar though. People mostly end up doing the first thing above with more advanced users probably delving into the second option (I know internally we have use cases of the second option).

I spent quite a bit of time experimenting with the first system yesterday and never got it to work. If I want to try the second, do you have documentation around it I can read through?

sorry was in meetings. Yes. That’s a good place to start but let me provide some additional context for the specific thing you want to do. It’s actually really really easy.

so would that work in the directory structure I provided before, or would this be a separate project?

you would probably have to make it look something like this:

main dir
|- dist
|- my_org_extensions
|  + metaflow_extensions
|     + org
|.       + my_utilities
| - projects dir
...

and you would have to, in your makefile: • build the metaflow_extensions package (and put it in dist) • install it so that it is available and installed when you do

myflow airflow …

(or whatever you do) For that last step, the basic thing is that this works if the package is “installed” (ie: when metaflow runs, it needs to find it). It can be installed as a package, in sys.path, etc but it needs to know about it.

gotcha. I'm thinking this is a fork/copy of the metaflow extensions project?

the metaflow-extensions repo you point to is indeed the “template” to use to make your own.

sounds like it will be a fun surprise then

in other words, it’s unlikely to break 🙂

out of curiosity, how do you share code to prevent repeats? I suppose you might not be able to share, but I see some threads around custom pypis, is that a "better" approach?

what do you mean “share code to prevent repeats”?

prevent repeated boilerplate code that's used across projects but is specific to our domain

the current method is what I am mentioning (extension package). INternally we have a group of DS that do a lot of the same thing so they standardized their own thing and use a metaflow extension to contain it all. That package is developed separately from all fllows/projects and users just install that package.

install it from a pypi?

yes

so this extension, installed via local pypi, allows for code to be placed on remote machines?

yes. Any extension code is automatically included in the package by metaflow

like, why use the extensions at all vs just using a local pypi?

metaflow packages itself and any extension installed.

well, they do have to install mygroup-metaflow

it replaces the installation of

metaflow

ahhhh

I mean, they have to install something

sure sure

metaflow is typically installed

ok so the workflow then, just to repeat so that I understand what's going on

yep. Said another way (to make sure I got it too): • team A works on

mycompany-metaflow

. Publishes to company pypi • team B who uses metaflow sets in their poetry toml file

mycompany-metaflow

instead of

metaflow

. THey get the benefit of all th shared stuff team A developed.

and the benefit of extending metaflow is to allow access to new decorators in the metaflow paradigm

the metaflow extension mechanism allows you to do a lot of things (although the deeper you go, the more risk you take we change things — it’s fairly stable though since as I said we use it internally). The simplest thing is what you started with (just package files) but yes, you can provide your own metaflow decorators that have hooks into the task lifecycle.

ok neat

so the remote instance doesn’t “install” anything.

gotcha. Question: if the extension library comes with all these caveats, why not just use an internal pypi? Won't metaflow package all my code and send it to remote machines as-is? Why expose the team to risk of the base extensions package changing?

metaflow packages all files in the directory of your flow file and below but nothing else. That is the 1st solution I mentioned.

ahhhh

yes — that is solution 1. As I mentioned, if you have non .py files, you need to do a few more things but yes. The code would be packaged in that way.

ah, I hadn't understood the "copy it lower into the hierarchy just so that it could be found", but that's definitely a

sys.path

solution and now I feel sheepish

always a win 🙂

thanks!