Hello everyone! I have a little problem when I try to access flow artifacts in a Jupyter Notebook, see below;

run_data = Flow('TestsDebug').latest_successful_run.data
run_data.config.config

error message:

368     # would not expect two artifacts with different names to actually
    369     # be aliases of one another)
--> 370     yield name, pickle.loads(blob)

ModuleNotFoundError: No module named 'modules'

I am running my flow on GCP with Kubernetes: My files and folders structure looks like this:

flows/
    TestsDebug.py
    modules/
            __init__.py
            config.py

My flow:

from metaflow import FlowSpec, step, kubernetes

class TestsDebug(FlowSpec):
    
    @kubernetes(image='my_image:latest')
    @step
    def start(self):
        from modules.config import Config
        self.config = Config()
        print('hello')
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    TestsDebug()

Any idea ?

You are pickling an object that relies on the modules thing in your code and that is not available to your Jupiter env.

Thanks Romain for your fast reply. Any idea how to fix this?

Yes. Sorry making food for kids. And yes, you need the same “environment” as when running. Placing the config.py file directly next to thr flow file will cause it to be packaged at the same time and in your jupyter env, if you have the config.py file there it will work.

I solved the issue by editing my local PYTHONPATH ;

PYTHONPATH=<path_to_folder>/flows/modules:<path_to_folder>/flows

Everything works.