What is the best way to share flows? For example I have some common flows in a package that people can reuse by specifying parameters to the flow. But to actually run the flow or create a step function, the user needs the path to the flow, which will be hidden somewhere in the depths of site-packages after they install my package as a dependency

would a shared repo work? I think that’s the commonest approach today

Shared repo doesn't really fit my use case because the parameters are all specified in config files which are bespoke to the user (so they'll have to check that into their own package)

certainly! You’d create a small wrapper that exposes the flows through a single command which you can distribute as a Python CLI package

do you have an example of how to import the flow and pass it the commandline arguments? when I try to create an instance of my flow i'm getting

OSError: could not get source code

you can just call it via

subprocess

as you would on the command line. Metaflow needs its own process anyways (and separate for child tasks), so a subprocess is needed in any case

i managed to make the cli and can invoke the flow locally. however when i deploy the flow to SFN it's not actually packaging up any of my local code and it can't find the flow file either. here is my setup:

package_A:
  module_A/
    flows/modelbuilder.py
    flows/module_A <- this is a symlink to ../../module_A
    cli.py
    other_modules/

package_B:
  module_B/
    more_modules

package_B has a dependency on package_A. package_A exposes

A-cli

which I invoke from package_B to run the

modelbuilder

flow with some params (which can reference modules in package_B using some import magic). The SFN is failing with the following error:

./__conda_python: can't open file '/metaflow/modelbuilder.py': [Errno 2] No such file or directory

There seems to be two problems: 1. modelbuilder.py doesnt exist <-- it comes from package_A which i do see in the

conda_v2.cnd

file as a dependency but the actual location will be somewhere in the depths of site-packages after it's installed 2. package_B code doesn't exist in the tarball Any suggestions/ideas on how to fix this or do what I want?