Hello, I’m designing a metaflow pipeline for sentiment analysis and during the pipeline initialization I encountered a problem. Suppose I have a file named

main_pipe.py

where my main pipeline exists. I have other files next to this file like

logger.py

and

utils.py

. When I want to import something (like a function) from these files. I do it like this.

from .utils import connect_to_db
from .logger import MAIN_LOGGER

Now when I run

python -m path.to.main_pipe run

I get python import error which indicates there is no parent module. Is there anyway to use other functions in other files in metaflow pipeline definition or the only solution is to include all the functionality in the

FlowSpec

class?

you can certainly use other packages and modules in addition to the main flow module. You need to have the main flow module, let's say

main_pipe.py

, at the top of the directory hierarchy like this:

main_pipe.py
utils/__init__.py
utils/some_util.py
logger/__init__.py

in

main_pipe.py

you can import the packages like this

from utils import connect_to_db
from logger import MAIN_LOGGER

and when you run the flow as

python main_pipe.py run

all packages get included automatically both for local and remote runs

Thanks that did the trick.