Hi there. I have a mono repo with a few shared code “libraries” in the root and a few projects all using those libraries. Now I want to create flows fro each project. Do I really need to add symlinks to each shared folder from each project, or is there a smarter way?? Maybe some kind of environment variable in which I can define the root from where to include everything?

Hi Helge, we have similar and we have just packaged these shared code ‘libraries’ as pip packages we can install (just using a simple setup.py) file , if you can package them all together, that would probably be most helpful. This certainly reduces complexity when it comes to building docker images etc for remote tasks.

that works this or another simple approach is to have the main flow files at the top of the directory hierarchy, so you don't need symlinks, like this:

/projects
   project_a_flow.py
   project_a/*
   project_b_flow.py
   project_b/*
   project_c_flow.py
   project_c/*
   shared_utils/*

@straight-shampoo-11124 Sorry to revive the dead here, is there a RX approach towards shared utils and shared libraries for multiple flows in a mono-repo?

“RX approach”? Not sure if it is useful knowledge or not, but we ended up with a solution where we are monkey patching / injecting a function into

MetaflowPackage.path_tuples

at runtime. This works very well for us and gives us full controle without having to implement a Metaflow plugins.

@thankful-father-61351 RX, common US slang for prescription or recommended. We ended up going with sym links in a mono repo with a submodule for the common_tools and it has worked pretty well thusfar

@clean-megabyte-14460 (sorry for a delayed reply - took a while to notice this old thread 🙂 ) did you check recommendations on this docs page?

@clean-megabyte-14460

it has worked pretty well thusfar

Seems fine in the beginning.. but then.. all of this 😉 https://outerbounds-community.slack.com/archives/C020U025QJK/p1694719262967809

Hahaha yeah I can see that problem coming. Did you figure out something more elegant from the two proposed solutions @thankful-father-61351?

No. But we did find a solution that works really well for us. We just override the

path_tuples()

function that Metaflow uses to find all the files to package. We do that by making sure this code is imported.

import itertools
import pathlib

from metaflow import metaflow_config
from metaflow import package
from metaflow.package import MetaflowPackage

INCLUDE_PATHS = []  # your shared code paths goes here
INCLUDE_SUFFIXES = [".py", ".yaml", ".yml", ".txt", ".cu", ".sh", ".json"]

def wrap_path_tuples(path_tuples_func):

    def wrapper(self):

        def list_files_in_paths(code_paths):
            for code_path in code_paths:
                for path in pathlib.Path(code_path).rglob("*"):
                    # Ignore hidden folders
                    if not str(path).startswith(".") and path.is_file() and path.suffix in INCLUDE_SUFFIXES:
                        yield str(path)


        tuples = itertools.chain(
            path_tuples_func(self),
            ((x, x) for x in list_files_in_paths(INCLUDE_PATHS)),
        )
        return tuples

    return wrapper

package.DEFAULT_SUFFIXES_LIST = INCLUDE_SUFFIXES
metaflow_config.DEFAULT_PACKAGE_SUFFIXES = ",".join(INCLUDE_SUFFIXES)
MetaflowPackage.path_tuples = wrap_path_tuples(MetaflowPackage.path_tuples)

A bit hacky, but we have not had any issues with this and we had a million issues before 🙂