RE Office Hours @pypi: @square-wire-39606 When developing and debugging code for internal packages locally, outside of the context metaflow, i would typically pip install --editable some packages. Would using @pypi be a suggested way to install local packages by a path in to the environment, or would you look to use a different method to include those packages/code in the execution environment? Create sym-links to the local packages to include as part of the code archive?

It’s probably easiest to include in the code package directly. The OSs pr does not have support as of now for local packages. You can check out the extension one that does (not in all cases — it won’t work across platforms for example and it’s not a very used path so there may be issues). In most cases though, if you are actively developing, it’s probably easiest to include your developed code directly but include thst packages dependencies in the conda environment. At least that would be what I would try first.

So far i've created a symlink in the same dir as my flow definition py file out to the source dir of that internal package, is that how you would include in the code package directly? or is there a better solution im not aware of yet

Yep. That’s how I would do it particularly if it is in active development. If you push that to a git repo, you could include the gitrepo in a requirements.txt with the extension and that would also work (in most cases).

An alternative (though more involved) approach: we have created a custom step decorator that uses the

add_to_package

lifecycle hook. So during local development we can decorate a step with

@package_local_module(modules=["local_import"])
def step():
    [...]

Nice!! That will work too :). That was initially added for cards. More advanced but definitely a nice option :).

That's very interesting @calm-energy-78015 we are also trying to find a way to get our local packages available for our flows/steps without having to use symlinks. We have a monorepo with several packages/folders holding several modules (.py). Is that your use case? or you are importing few modules per step? Would you mind sharing how you defined your decorator?

@fast-pizza-24629 we have a monorepo, but typically only working on 1 or 2 packages at any one time. Simplified version:

from metaflow.decorators import StepDecorator

class PackageLocalModules(StepDecorator):
    name = "package_local_modules"
    defaults = {"modules": []}

    def add_to_package(self):
        from importlib.util import find_spec
        from pathlib import Path

        package_files = []
        for module in self.attributes["modules"]:
            module_root = Path(find_spec(module).origin).parent
            package_files.extend(module_root.glob("**/*"))

        return package_files

we have some extra error handling, and functionality to ignore certain file/directory types, but that's the essential bit. I'd previously said it was a flow decorator, but it is actually a step decorator. We have created a

decorate_all_steps

flow decorator that we use to only specify it once per flow.

Nice, thank you for sharing. Yeah, I was looking into the stepDecorators as well 🙏