is it ever valid to create a flow inside of a package? ex: if my directory structure is like:

package_dir/
    __init__.py
    flows.py
    shared_code.py

And say that

flows.py

looks something like

from metaflow import FlowSpec, step

from package_dir import shared_code
 
 
class ExampleFlow(FlowSpec):
    ...
 
if __name__ == "__main__":
    ExampleFlow()

if I try to run this like this:

python -m package_dir.flows run

I get a

ModuleNotFoundError

when trying to import the

shared_code

module, but that import works when I comment out the

ExampleFlow()

call and replace it with something like

print("Hello world!")

,

is there a reason you need to use the -m option?

usualyl when I run into a similar issue with other scripts, using

-m

helps it figure out the imports

yeah its likely because you're running it from the directory above

hm

raises an interesting point though, since metaflow is so flexible there's not really a clear right/wrong way to do anything

I wonder if there's a way to organize all the `FlowSpec`s in their own directory and just have a single file that runs them

even if it was a larger project I'd probably have a directory structure that runs flows from the same directory:

./
 flows/
   flow_1.py
   other_flow.py
     common/
       __init__.py
       utils.py
 infra/
   src/ 
   stack.ts
...

I see

and when I'm actually working in on the flows I'm just in that directory so it works

I guess I generally prefer to have stuff organized like

project_code/
    __init__.py
    subpackage/
        __init__.py
        functions.py
    submodule.py

flows/
    __init__.py
    flow1.py
    flow2.py

as in my head, I tend to think of these flows as "this is how I want to orchestrate my code" and the stuff under

project_code

as "this is the business logic that drives the pipelines". This is a minor point, but it also helps having these in separate directories because then in a file explorer or whatever I can separately collapse the

flows

directory or the

project_code

directories if I want to limit my focus to a certain set of files

at that point you could just create a proper package out of it and have cli commands to run the flows (basically the makefile approach)

like package

project_code

and all the stuff under

flows

can treat it like any other package in

site-packages

?

and then it should work if you have the _init_s set up properly - accidental automatic italicization there, imagine there’s __’s there

haha yeah I know what you mean